DataBricks Tech Lead

Tech Lead - DataBricks

As a Tech Lead, you will work closely with business stakeholders, solution architects, product owner, scrum master, and subject matter experts (SMEs) to understand requirements and deliver high-quality, scalable solutions on Azure Databricks.

Responsibilities Summary:
•Lead solution design and delivery for Databricks-based platforms and products (batch/streaming, BI-ready models, ML workflows).
•Translate business requirements into reference architectures, implementation plans, and sprint-ready technical stories.
•Serve as technical decision-maker for tradeoffs (cost, performance, latency, reliability, maintainability).
•Design and implement Lakehouse patterns (Bronze/Silver/Gold), medallion architecture, and domain-oriented data products where applicable.
•Build and optimize ETL/ELT using Apache Spark, Databricks SQL, and orchestrators (e.g., Workflows, ADF, Airflow).
•Implement streaming use cases (e.g., Spark Structured Streaming, Delta Live Tables where appropriate).
•Establish data modeling standards (star/snowflake, Data Vault where relevant) and performance tuning practices.
•Implement access controls, auditing, and governance with Unity Catalog (RBAC/ABAC patterns, lineage, data sharing policies).
•Ensure production readiness: CI/CD, monitoring/alerting, runbooks, incident response, and SLAs/SLOs.
•Drive data quality practices (tests, expectations, reconciliation, observability).
•Define MLOps standards (experiment tracking, reproducibility, champion/challenger, drift monitoring).
•Mentor engineers; conduct design reviews, code reviews, and set engineering standards.
•Partner with product owners, data owners, security, and platform teams; communicate status, risks, and options clearly.
•Contribute to hiring, onboarding, and capability building.

Position Requirements:
•7+ years in data/platform/analytics engineering, including 2+ years leading technical teams or workstreams.
•Proven production delivery on Databricks (with strong Azure Databricks experience preferred).
•Strong Apache Spark expertise (PySpark/Scala): distributed processing, troubleshooting, and performance tuning.
•Deep Delta Lake knowledge: ACID tables, compaction, Z-Ordering, schema evolution, and batch/streaming patterns.
•Experience building scalable batch and streaming pipelines, including orchestration/operationalization (scheduling, dependencies, retries, idempotency).
•Strong Azure data platform background, including Azure Data Lake Storage (ADLS) architecture and best practices; familiarity with Azure Data Factory (ADF), Azure Synapse Analytics, and related services.
•Advanced programming skills in Python and SQL, plus hands-on Java experience (e.g., integrations/services or Spark/platform utilities).
•Cloud fundamentals in at least one major platform (Azure/AWS/Google Cloud Platform (Google Cloud Platform)): identity and access management (IAM), storage, networking basics, and cost controls.
•Working experience in Java (e.g., building integrations/services, Spark/streaming components, or platform utilities).