Databricks Architect

Responsibilities & Skills:

·         12 YRS Exp

·         Design and implement enterprise-scale solutions on the Databricks Lakehouse Platform.

·         Architect end-to-end data pipelines for batch and real-time processing using Apache Spark and PySpark.

·         Develop scalable data ingestion, transformation, and data quality frameworks.

·         Design and implement Medallion Architecture (Bronze, Silver, Gold) using Delta Lake.

·         Build and optimize data warehouses, data marts, and analytical solutions.

·         Implement data governance, security, lineage, and access controls using Unity Catalog.

·         Develop and support AI/BI dashboards, semantic models, and self-service analytics solutions.

·         Configure and optimize Genie Spaces to enable natural language business queries and conversational analytics.

·         Design and deploy Generative AI and RAG-based solutions using Databricks Mosaic AI and Vector Search.

·         Collaborate with business users to translate requirements into scalable data and AI solutions.

·         Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency.

·         Lead cloud-native implementations across  Azure environments.

·         Define architecture standards, best practices, and reusable design patterns.

·         Mentor data engineers, analysts, and architects on Databricks technologies and platform adoption.

·         Lead migration initiatives from legacy data warehouses and analytics platforms to Databricks.

·         Build and maintain Genie Spaces for business self-service analytics.

·         Create semantic models, metrics, and trusted data assets for AI-driven reporting.

·         Develop natural language-to-SQL analytics solutions using Databricks Genie.

·         Implement RAG solutions using enterprise data and Vector Search.

·         Optimize AI/BI dashboards and conversational analytics experiences.

·         Troubleshoot Spark performance, query optimization, and workload management.

·         Automate data validation, monitoring, and governance controls.

·         Support AI use cases using Mosaic AI model serving and inference endpoints.

Technical Skills

·         Databricks Lakehouse Platform

·         Apache Spark, PySpark, Spark SQL

·         Python, SQL

·         Delta Lake, Delta Live Tables, Lakeflow

·         Unity Catalog

·         Databricks AI/BI and Genie

·         Mosaic AI, Vector Search, RAG

·         Data Modeling (Dimensional & Data Vault)

·         Structured Streaming

·         Data Quality and Data Governance

·         Azure

·         Terraform, Git, Azure DevOps, Jenkins

·         REST APIs and Data Integration

·         Performance Tuning and Cost Optimization

Certification : Azure Databricks certified Data Eng professional