Databricks Data Engineer

Position: Databricks Data Engineer

Location: Austin, TX (Preferred Locals to Austin, TX and Permanent residents of USA)

Duration: Long-Term Contract

Key Skills: Looking for a Senior Databricks Data Engineer with SQL Server + SSIS + DW modernization experience, and ideally GenAI/LLM/Agent experience

Responsibilities & Skills:

• 12yrs of Data Engineer and IT exp is mandatory
• Design and implement enterprise scale solutions on the Databricks Lakehouse Platform.
• Architect end to end data pipelines for batch and real time processing using Apache Spark and PySpark.
• Develop scalable data ingestion, transformation, and data quality frameworks.
• Design and implement Medallion Architecture (Bronze, Silver, Gold) using Delta Lake.
• Build and optimize data warehouses, data marts, and analytical solutions.
• Implement data governance, security, lineage, and access controls using Unity Catalog.
• Develop and support AI/BI dashboards, semantic models, and self-service analytics solutions.
• Configure and optimize Genie Spaces to enable natural language business queries and conversational analytics.
• Design and deploy Generative AI and RAG based solutions using Databricks Mosaic AI and Vector Search.
• Collaborate with business users to translate requirements into scalable data and AI solutions.
• Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency.
• Lead cloud native implementations across Azure environments.
• Define architecture standards, best practices, and reusable design patterns.
• Mentor data engineers, analysts, and architects on Databricks technologies and platform adoption.
• Lead migration initiatives from legacy data warehouses and analytics platforms to Databricks.
• Build and maintain Genie Spaces for business self-service analytics.
• Create semantic models, metrics, and trusted data assets for AI driven reporting.
• Develop natural language to SQL analytics solutions using Databricks Genie.
• Implement RAG solutions using enterprise data and Vector Search.
• Optimize AI/BI dashboards and conversational analytics experiences.
• Troubleshoot Spark performance, query optimization, and workload management.
• Automate data validation, monitoring, and governance controls.
• Support AI use cases using Mosaic AI model serving and inference endpoints.

Technical Skills:

• Databricks Lakehouse Platform
• Apache Spark, PySpark, Spark SQL
• Python, SQL
• Delta Lake, Delta Live Tables, Lakeflow
• Unity Catalog
• Databricks AI/BI and Genie
• Mosaic AI, Vector Search, RAG
• Data Modeling (Dimensional & Data Vault)
• Structured Streaming
• Data Quality and Data Governance
• Azure
• Terraform, Git, Azure DevOps, Jenkins
• REST APIs and Data Integration
• Performance Tuning and Cost Optimization