Key Responsibilities
Design and implement enterprise-wide data quality frameworks aligned with Lakehouse architecture, including Bronze, Silver, and Gold data layers
Define, implement, and enforce data quality rules covering completeness, accuracy, consistency, timeliness, and validity
Develop reusable validation, reconciliation, profiling, and monitoring frameworks within Databricks environments
Establish automated data quality checks integrated into ELT and ETL pipelines
Embed quality controls directly into Databricks workflows, Spark processing pipelines, and Delta Lake architectures
Develop scalable validation processes supporting both batch and real-time data ingestion pipelines
Partner with Data Engineers to ensure quality gates are enforced across ingestion, transformation, and consumption layers
Optimize data quality processes for performance, scalability, and reliability across large distributed datasets
Implement and maintain data observability solutions, including dashboards, alerts, monitoring metrics, and reporting frameworks
Monitor data pipelines and proactively identify anomalies, failures, data drift, and data quality degradation
Lead root cause analysis (RCA) activities and drive resolution of data quality issues
Develop and maintain enterprise data quality scorecards and performance reporting
Ensure adherence to enterprise data governance standards, including metadata management, data lineage, traceability, and auditability
Collaborate with Data Governance teams to align data definitions, ownership models, and control frameworks
Support regulatory compliance requirements through auditable and repeatable data quality processes
Define and enforce data quality SLAs, standards, and data contracts across business domains
Implement CI/CD practices for data quality rules, monitoring processes, and validation frameworks
Automate testing and validation of data transformations, integrations, and pipelines
Develop reusable enterprise libraries and frameworks for scalable data quality enforcement
Partner with Data Architects, BI teams, Data Engineers, and business stakeholders to drive data quality initiatives
Provide technical leadership, mentorship, and best-practice guidance across teams
Serve as the subject matter expert (SME) for enterprise data quality strategies and standards
Drive continuous improvement and innovation in data quality methodologies, tools, and practices
Required Qualifications
5+ years of experience
Bachelor's degree in Computer Science, Data Engineering, Information Systems, Data Science, or a related field
Databricks Certified Data Engineer Associate or Professional
Databricks Certified Developer for Apache Spark
AWS Certified Data Engineer Associate
Microsoft Certified: Azure Data Engineer Associate
Collibra Data Governance Certification
Relevant Data Quality, Data Governance, or Cloud Certifications
Skills
Databricks
PySpark
Delta Lake
SQL
Apache Spark
ETL/ELT Pipeline Development
Data Validation and Reconciliation
Cloud Data Platforms (AWS or Azure)
Metadata Management and Data Lineage
CI/CD for Data Quality
Data Quality Tools (Great Expectations, Deequ, Monte Carlo)
Collibra
Streaming Data Quality Validation
Machine Learning-Driven Data Quality Monitoring
Data Lifecycle Management
Data Quality Framework Implementation
Data Governance
Data Observability
Root Cause Analysis
Stakeholder Management
Technical Leadership
Mentoring and Coaching
Requirements Gathering
Regulatory Compliance
Data Contracts and Enterprise Data Standards
Agile Delivery
Schedule
Start date: 2026-09-18
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: compun
- Position Id: SHADC5888810
- Posted 3 hours ago
Never miss an opportunity! Create an alert based on the job you applied for.
