Databricks SME

Job Title: Databricks SME

Role Summary

We are seeking a highly skilled and experienced Databricks SME to lead the architecture, design, and implementation of enterprise-scale Big Data solutions on the Databricks Lakehouse Platform. The ideal candidate will have at least 12 years of hands-on experience architecting big data solutions with Databricks, with proven expertise in migrating data and pipelines from legacy platforms like Alteryx, handling streaming data, and implementing CI/CD practices for data pipelines.

This role is critical for driving data modernization initiatives, ensuring robust data governance, and enabling scalable data ingestion and transformation across multiple source systems into the Databricks ecosystem.

Key Responsibilities

• Architect enterprise Big Data solutions on Databricks using Bronze/Silver/Gold (Medallion) layers with Delta Lake and Delta Live Tables.

• Migrate data and pipelines from legacy platforms (Alteryx, Informatica, DataStage, Talend etc.) to Databricks, including source-to-target mapping and data validation.

• Design streaming data pipelines using Structured Streaming, Auto Loader, and Delta Live Tables.

• Implement CI/CD practices for data pipelines using Git, Azure DevOps, and GitHub Actions.

• Develop data governance framework using Unity Catalog ABAC/RBAC policies.

• Collaborate with stakeholders to translate requirements into scalable solutions.

• Experience with Databricks Workflows, Jobs, and Databricks Asset Bundles for deployment automation.

• Mentor junior team members and provide technical leadership across data engineering projects.

Required Skills

• 12+ years architecting Big Data solutions with Databricks Lakehouse Platform.

• Proven experience migrating pipelines from Alteryx or similar legacy ETL tools.

• Hands-on expertise with Delta Live Tables, Delta Lake, and Medallion architecture.

• Strong experience building streaming pipelines with Structured Streaming and Auto Loader.

• Demonstrated ability implementing CI/CD for data workloads.

• Unity Catalog Data Governance framework with ABAC/RBAC policies.

• Proficient in Python, SQL, PySpark.

• Strong experience in performance tuning of large-scale distributed pipelines.

• Proven experience on at least one major cloud platform — Azure (preferred) or AWS.

Preferred Skills

• Unity Catalog for data governance and lineage.

• Data Quality tools: Great Expectations, Informatica DQ, Collibra.

• Data Modeling tools: Erwin, dbt, ER/Studio.

• AI/ML experience with GenAI, LLMs, MLOps.

• Azure certifications in Data Engineering.

Certifications (Preferred)

• Databricks Certified Data Engineer Associate/Professional.

• Microsoft Certified: Azure Data Engineer Associate.

Experience Required

12+ Years of Databricks/Big Data Architecture experience.