Sr Data Quality Engineer

Job Title: Sr Data Quality Engineer

Duration: Long Term - Can be extended

Location: Chicago, IL.

Job Description:

Data Quality Engineering & Frameworks

• Design and implement enterprise-wide data quality frameworks aligned to Lakehouse architecture (bronze, silver, gold layers)
• Define and enforce data quality rules including completeness, accuracy, consistency, timeliness, and validity
• Develop reusable data validation, reconciliation, and monitoring patterns within Databricks pipelines
• Establish automated data quality checks embedded within ELT/ETL workflows

Databricks & Pipeline Integration

• Integrate data quality controls directly into Databricks (Spark/Delta Lake) pipelines and workflows
• Develop scalable validation processes for batch and event-driven ingestion pipelines
• Partner with Data Engineers to ensure quality gates are enforced across ingestion, transformation, and consumption layers
• Optimize data quality processes for performance and scalability within large distributed datasets

Monitoring, Observability & Issue Management

• Implement and manage data observability frameworks, including metrics, alerts, and dashboards
• Monitor data pipelines and proactively identify anomalies, failures, and quality degradation
• Lead root cause analysis (RCA) efforts for data quality issues and drive remediation
• Develop and maintain quality scorecards and reporting for stakeholders

Data Governance & Compliance

• Ensure adherence to enterprise data governance standards, including metadata, lineage, and auditability
• Partner with Data Governance teams (e.g., Collibra) to align data definitions, ownership, and controls
• Support regulatory requirements (e.g., SOX, GLBA, data integrity standards) through auditable data quality controls
• Define and enforce data quality SLAs and data contracts across domains

Automation & DevOps

• Implement CI/CD practices for data quality rules, validations, and monitoring
• Automate testing frameworks for validating data transformations and pipelines
• Develop reusable libraries and frameworks for enterprise-scale data quality enforcement
• Collaboration & LeadershipPartner with Data Engineers, Data Architects, BI teams, and business stakeholders to embed quality-by-design principles
• Provide technical leadership and mentorship on data quality best practices
• Act as a subject matter expert (SME) for data quality across the organization
• Drive continuous improvement and innovation in data quality tooling and methodologies
• Required Qualifications5+ years of experience in data engineering, data quality engineering, or related roles
• Strong hands-on experience with Databricks, Spark (PySpark), and Delta Lake
• Proven experience implementing data quality frameworks and controls in modern data platforms
• Advanced SQL and data profiling/validation skills
• Experience working with large-scale datasets in cloud environments (AWS or Azure)
• Experience integrating data quality into ELT/ETL pipelines and orchestration tools
• Strong understanding of data governance and data lifecycle management

Preferred Qualifications

• Experience in financial services or regulated environments
• Familiarity with data governance tools (e.g., Collibra)
• Experience with data observability or quality tooling (e.g., Monte Carlo, Great Expectations, Deequ, or similar)
• Experience with real-time data quality validation (streaming pipelines)
• Knowledge of regulatory reporting and data controls frameworks
• Cloud or Databricks certifications

Technical Skills

• Databricks (Lakehouse, Unity Catalog, workflows)
• Spark / PySpark
• SQL (advanced)
• Delta Lake
• Data quality frameworks (rule engines, validation patterns)
• Data observability and monitoring
• Cloud platforms (AWS or Azure)
• Orchestration tools (Airflow, Control-M)
• APIs and data integration
• CI/CD and DevOps
• Data modeling and lineage concepts

Professional Competencies

• Strong analytical and problem-solving skills with a focus on data integrity
• High attention to detail and commitment to data accuracy
• Strong communication skills across technical and non-technical stakeholders
• Ability to influence standards and drive enterprise adoption
• Collaborative mindset with a focus on continuous improvement