Senior Databricks Architect

Job Summary

• We are seeking a highly experienced Databricks Architect to lead the design, development, and modernization of enterprise-scale data platforms.
• The ideal candidate will have deep expertise in Databricks Lakehouse Architecture, Spark, Delta Lake, Unity Catalog, Data Governance, and cloud-native data engineering solutions across Azure, AWS, or Google Cloud Platform.
• This role requires strong architecture, leadership, and hands-on implementation experience supporting large-scale analytics, AI/ML, and real-time data processing initiatives.

Key Responsibilities

• Architect enterprise-scale Lakehouse solutions using Databricks.
• Design and implement scalable batch and streaming data pipelines.
• Establish enterprise standards for Delta Lake, Unity Catalog, governance, lineage, and security.
• Lead cloud migration initiatives from legacy/on-prem platforms to Databricks.
• Build and optimize Medallion Architecture (Bronze/Silver/Gold).
• Implement CDC, data quality frameworks, and metadata management solutions.
• Drive performance tuning, cluster optimization, and cloud cost reduction strategies.
• Collaborate with Data Engineers, Data Scientists, ML Engineers, and business stakeholders.
• Design scalable AI/ML and GenAI-ready data platforms.
• Mentor engineering teams and provide technical leadership across multiple projects.

Required Experience

• 12+ years of overall IT experience.
• 8+ years in Data Engineering and Big Data ecosystems.
• 5+ years of hands-on Databricks architecture and implementation experience.
• Proven experience leading enterprise data modernization and cloud migration programs.
• Strong client-facing and stakeholder management skills.

Core Technical Skills

• Databricks Lakehouse Platform.
• Apache Spark (PySpark/Scala).
• Delta Lake & Delta Live Tables (DLT).
• Unity Catalog & Data Governance.
• Structured Streaming & Real-Time Data Processing.
• SQL, Python, Spark SQL.
• Azure Databricks / AWS Databricks / Google Cloud Platform Databricks.
• Medallion Architecture (Bronze, Silver, Gold).
• Data Modeling (Dimensional, Star Schema, Data Vault).
• CI/CD, DevOps, GitHub Actions, Azure DevOps.
• Terraform, Infrastructure as Code (IaC).
• Airflow, Databricks Workflows.
• Machine Learning & MLOps.
• Generative AI and AI/ML Enablement on Databricks.
• Performance Tuning & Cost Optimization.