Job Title:Senior Data Engineer, Azure Databricks
Location:Fully Remote (on-site in Dallas, Memphis, Cleveland, or Pittsburgh is optional)
Employment Status:Applicants must be currently authorized to work in the U.S. without the need for employment-based sponsorship, now or in the future, including: H-1B, L-1, TN, O-1, E-3, H-1B1, F-1, J-1, OPT, CPT, or any other employment-based visa programs.
Position Overview
The Senior Data Engineer will design, develop, and optimize enterprise data warehouse solutions and production data pipelines within an Azure Databricks environment. This hands-on position will support the continued development and modernization of the data ecosystem including the transition toward real-time, event-driven data processing, while ensuring data quality, reliability, and performance in a highly regulated environment.
Key Responsibilities
Design, develop, and maintain enterprise data warehouse solutions and production pipelines across Bronze, Silver, and Gold layers using Azure Databricks, Delta Lake, and medallion architecture. Develop and optimize data ingestion, transformation, and modeling solutions using SQL, Python, PySpark, and Azure Data Factory. Build dimensional and harmonized data models, incorporating data conformance, survivorship logic, and standardized transformation processes. Support the transition from traditional batch processing toward continuous, low-latency, event-driven data ingestion using Azure Event Hubs, Structured Streaming, and emerging Databricks capabilities. Own assigned production data domains, ensuring pipeline reliability, data integrity, performance, and compliance with established service-level requirements. Monitor and optimize pipeline performance, processing latency, scalability, and Databricks compute costs. Troubleshoot complex production issues, perform root-cause analysis, and implement sustainable improvements to data reliability and system performance. Implement automated data validation, reconciliation, quality controls, and testing practices throughout the data lifecycle. Support data governance initiatives involving Unity Catalog, metadata management, lineage, access controls, and Gold-layer data assets. Modernize and consolidate legacy transformation processes while maintaining data accuracy and operational continuity. Establish and maintain engineering best practices for code quality, testing, documentation, version control, and CI/CD deployment within a highly regulated environment. Collaborate with Data Architecture, Analytics Engineering, QA, Data Governance, and business stakeholders to deliver reliable, scalable data solutions. Develop and enhance internal and external partner data integrations to support evolving business and operational requirements. Provide technical guidance and mentorship to Data Engineers through code reviews, knowledge sharing, and engineering best practices.
Qualifications 6+ years of progressive data engineering experience including hands-on development and production support of enterprise data platforms. Deep expertise in Azure Databricks, Delta Lake, Medallion Architecture, Unity Catalog, and production pipeline development and optimization. Hands-on experience developing and operating event-driven streaming data pipelines using Azure Event Hubs, Kafka, AWS Kinesis, or Confluent Cloud. Strong enterprise data warehousing experience encompassing dimensional modeling, complex transformations, data integration, and production operations. Advanced proficiency in SQL, Python, and PySpark, with the ability to independently develop, troubleshoot, review, and optimize complex data pipelines. Hands-on experience with Azure Data Lake Storage, Azure Data Factory, and CI/CD practices using Azure DevOps. Demonstrated ownership of production data pipelines, including monitoring, incident response, performance optimization, and service-level accountability. Experience implementing data contracts using ODCS or equivalent standards preferred. Experience implementing data validation, reconciliation, automated testing, and data quality controls. Strong understanding of data governance, metadata management, lineage, and secure data handling practices. Familiarity with data observability platforms such as Monte Carlo preferred. Experience operating within highly regulated environments including healthcare, pharmacy, pharmaceutical, medical, or another highly regulated industry involving sensitive clinical or operational data is preferred. Ability to analyze complex legacy data environments, improve existing architectures, and produce comprehensive technical documentation. Familiarity with Structured Streaming, Zerobus Ingestion, and Spark Declarative Pipelines preferred. Experience with enterprise data catalog platforms such as Atlan, Collibra, or equivalent technologies preferred. Experience developing governed data assets with documented metadata, lineage, and access controls. Strong communication, problem-solving, and cross-functional collaboration skills.
