COOLSOFT
COOLSOFT

Senior Databricks Data QA Engineer/Data Engineering

Remote • Posted 2 hours ago • Updated 2 hours ago
Contract W2
12 Months
Occasional Travel Required
Remote
$60 - $70/hr
Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Job Details

Skills

  • Azure Databricks
  • Data Quality Assurance
  • Data Engineering
  • Python
  • PySpark
  • Spark SQL
  • Azure Data Factory (ADF)
  • ETL/ELT Pipeline Testing
  • Delta Lake
  • Delta Live Tables (DLT)
  • Databricks Workflows
  • Unity Catalog
  • Azure DevOps
  • CI/CD
  • Test Automation
  • Data Quality Frameworks
  • Healthcare Payer Data Validation
  • FHIR API Testing
  • HEDIS Validation
  • Data Governance
  • HIPAA Compliance
  • CMS Interoperability
  • PII Compliance
  • Load and Stress Testing
  • Technical Leadership
  • QA Mentoring
  • Databricks DQX (Preferred)
  • Databricks Asset Bundles (DAB) (Preferred)

Summary

 

Title:Data Engineering

mployment Type: Three-month contract-to-hire; full-time conversion expected after three months

Position Summary

We are seeking a Senior Databricks Data QA Lead with extensive data engineering experience to lead data quality assurance and automation across healthcare payer data platforms. This role will build automated validation frameworks, establish deployment quality gates, and ensure the accuracy, reliability, performance, and compliance of enterprise data pipelines.

Key Responsibilities

  • Design and implement test plans, test cases, and automated validation for ETL/ELT pipelines.
  • Validate healthcare payer datasets, including claims, membership, provider, and clinical data.
  • Conduct FHIR-based API testing to support CMS interoperability compliance.
  • Validate HEDIS measure calculations and quality reporting data.
  • Build and maintain data quality automation frameworks using Python, PySpark, and Spark SQL.
  • Develop Azure DevOps test plans and QA gates for Databricks Jobs, Delta Live Tables, and Azure Data Factory deployments.
  • Collaborate with product owners, data engineers, governance teams, and business stakeholders to identify and resolve data quality issues.
  • Design automated load and stress tests for large-scale data pipelines.
  • Ensure data QA practices adhere to HIPAA, CMS, PII protection, and payer industry requirements.
  • Document QA processes, test results, and audit-ready evidence.
  • Mentor and guide junior QA engineers.

Required Qualifications

  • 12+ years of Data Engineering experience.
  • 5+ years working exclusively on Databricks.
  • Strong expertise in Azure Databricks, including Workflows, Delta Lake, Delta Live Tables, and Unity Catalog.
  • Hands-on experience with Azure Data Factory pipelines, monitoring, and CI/CD.
  • Proficiency in Python, PySpark, and Spark SQL for data quality automation.
  • Experience validating healthcare payer data and working with FHIR APIs and HEDIS reporting.
  • Experience with Azure DevOps, automated deployment pipelines, and QA gates.
  • Knowledge of data governance, PII compliance, and data quality frameworks.
  • Experience testing the performance and scalability of data pipelines.
  • Strong communication, problem-solving, and technical leadership skills.
  • Willingness to convert to full-time employment after the initial contract period.

Preferred Qualifications

  • Experience with Databricks DQX.
  • Experience with Databricks Asset Bundles (DAB).
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: nm13509
  • Position Id: 9132136
  • Posted 2 hours ago
Contact the job poster
KJ

Krishnendu Jana

Recruiter @ COOLSOFT

View Profile
Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.
Remote jobs at COOLSOFT