Data Quality Manager

JOB DESCRIPTION

Technical Lead - Azure Databricks

Technical Leadership & Architecture

• Serve as the primary technical authority for Azure Databricks solution design, architecture decisions, and implementation standards.
• Define and enforce Medallion/Lakehouse (Bronze/Silver/Gold) architecture patterns, Delta Lake schema design, Unity Catalog governance, and cluster configuration best practices.
• Drive design reviews and technical trade-off analyses, ensuring alignment with client objectives, timelines, and compliance requirements.

Team Leadership & People Management

• Lead 7+ Senior Data Engineers with day-to-day technical direction, prioritization, and removal of blockers.
• Conduct rigorous code reviews ensuring adherence to coding standards, performance best practices, and maintainability guidelines.
• Mentor senior engineers on Databricks features, python/spark optimization, data modeling, and software engineering excellence.
• Collaborate with the Senior Engineering Manager on effort estimates, risk identification, sprint planning, and stakeholder communications.

Ingestion Pipeline Design & Delivery

• Architect and oversee delivery of all ingestion pipelines across three core patterns: on-premises SQL Server (JDBC/linked services, CDC), REST/SOAP API integrations (pagination, auth, rate-limiting), and flat file ingestion (CSV, JSON, XML, SFTP/blob-based landing zones).
• Establish reusable, metadata-driven ingestion frameworks in PySpark and Python that support all three patterns with consistent error handling, logging, and retry logic.
• Ensure robust integration with ADLS Gen2, Azure Key Vault, and Azure DevOps across all ingestion workflows.
• Champion Delta Lake best practices: schema evolution, incremental loads, partition strategies, and optimize/ZORDER tuning.
• Lead CI/CD pipeline implementation, automated testing, and security scanning for reliable, repeatable deployments.

Data Quality, Governance & Compliance

• Establish data quality frameworks (Great Expectations or DQX) covering validation, automated monitoring, SLA alerting, and lineage tracking across all ingestion patterns.
• Lead Unity Catalog adoption including workspace access controls, column-level security, and audit logging; ensure compliance with legal data confidentiality and privilege-protection requirements.

Performance, Optimization & Troubleshooting

• Own platform performance: diagnose and resolve Spark, Delta Lake, and cluster issues; optimize job runtimes, resource utilization, and cost efficiency.
• Set monitoring and observability standards using Databricks system tables and Azure Monitor; serve as escalation point for critical production issues.

Required

• Bachelor's degree in Computer Science, Engineering, Data Science, or a related field.
• 8+ years of hands-on data engineering experience including large-scale ETL/ELT and Lakehouse platform delivery.
• 3+ years on Databricks including production deployments, cluster management, and enterprise integrations.
• Demonstrated experience technically leading teams of senior engineers, driving architecture direction and code quality.
• Deep expertise in Databricks components: Delta Lake, Databricks SQL, Apache Spark, Unity Catalog, Databricks Workflows, and Notebooks.
• Proven experience designing and building ingestion pipelines across all three patterns: on-premises SQL Server (JDBC, CDC), REST/SOAP APIs (auth flows, pagination, rate-limiting), and flat files (CSV, JSON, XML via blob storage or SFTP).
• Advanced proficiency in PySpark, Spark SQL, and Python; working knowledge of Scala.
• Strong Azure ecosystem experience: ADLS Gen2, Azure Data Factory, Azure DevOps, Azure Key Vault, and Azure Monitor.
• Proven track record establishing engineering best practices: code reviews, CI/CD, test automation, and documentation standards.
• Strong data governance and compliance understanding including RBAC, data classification, and legal data confidentiality requirements.
• Excellent communication skills with ability to engage both engineering teams and non-technical legal stakeholders.

Preferred

• "Proficiency with VS Code for development workflows including Python, Databricks Connect, and remote development extensions; familiarity with Claude Code for AI-assisted coding, code review support, and accelerating engineering productivity."
• Databricks Certified Data Engineer Professional or Microsoft Certified: Azure Data Engineer Associate or equivalent.
• Master's degree in Computer Science, Engineering, or a related field.
• Experience with data quality frameworks such as Great Expectations or Databricks DQX for automated validation and expectation suites within Databricks pipelines.
• Hands-on experience with Infrastructure as Code (IaC) using Terraform or Bicep for Azure resource provisioning.
• Familiarity with enterprise data modeling tools (e.g., ERwin) for logical and physical data model interpretation.
• Experience migrating legacy on-premises data warehouses or SQL Server environments to Azure Databricks Lakehouse architectures.
• Exposure to ML workloads on Databricks (MLflow, Feature Store) and feature engineering pipeline collaboration.
• Familiarity with Agile/Scrum delivery using Azure DevOps Boards or Jira.