job summary:
We are seeking Senior Data Engineers for a 100% remote role supporting the modernization of a healthcare pharmacy compliance platform using Databricks. Working within a delivery pod under a technical lead, you will build pipelines, transformations, and validation tools for high-volume ingestion and processing on Delta Lake while re-implementing business logic from SQL Server.
The platform supports high-stakes financial and regulatory decisions. Accuracy takes priority over throughput, with an emphasis on testing, reconciliation against legacy systems, and traceability in engineering decisions.
location: Telecommute
job type: Solutions
salary: $75 - 82 per hour
work hours: 8am to 5pm
education: Bachelors
responsibilities:
- Build and maintain ingestion pipelines for high-volume batch and intraday feeds into a Delta Lake architecture with bronze, silver, and gold layers.
- Implement transformations and business logic in PySpark and Spark SQL using established rule specifications.
- Translate and re-implement SQL Server stored procedure logic in Spark, identifying constructs that do not translate cleanly.
- Build data quality checks, validation rules, and automated tests, including tests that compare new results with legacy output.
- Support reconciliation through comparison jobs, discrepancy reporting, and triage tools.
- Optimize Spark performance through partitioning, file sizing, join strategies, skew handling, and cluster configuration while controlling compute costs.
- Create idempotent, re-runnable pipelines that handle incremental and late-arriving data.
- Participate in code reviews, follow engineering standards, and contribute to project documentation.
- Collaborate with the technical lead and subject matter experts to clarify requirements and resolve ambiguity.
qualifications:
Qualifications
Required:
6+ years of data engineering experience, including 3+ years of production experience with Databricks and Apache Spark.
Strong Python and PySpark skills, with the ability to independently write, test, and tune production code.
Proficiency in SQL and T-SQL, including interpreting and translating existing stored procedures.
Practical Delta Lake experience, including MERGE/upsert operations, incremental loads, partitioning, time travel, schema evolution, and table maintenance.
Deep understanding of the Spark execution model, including jobs, stages, shuffles, joins, and partitioning, with the ability to troubleshoot performance issues.
Experience building idempotent, re-runnable pipelines and managing late-arriving or corrected data.
Proficiency with Git, automated testing, and CI/CD for data pipelines.
Strong written communication skills.
Preferred:
Experience with healthcare, pharmacy, payer, or life sciences data, such as HL7, ADT, ePrescribe, NCPDP, claims, NDC drug data, or HIPAA-regulated PHI.
Experience migrating SQL Server workloads to a lakehouse or Databricks.
Familiarity with Azure Data Factory, Azure Storage, and the broader Azure data stack.
Experience with Delta Live Tables, Databricks Workflows, and Unity Catalog.
Knowledge of structured streaming or incremental micro-batch processing.
Familiarity with dbt or comparable transformation frameworks.
Experience with data quality frameworks such as Great Expectations.
Databricks Certified Data Engineer Associate or Professional certification.
Exposure to record linkage, entity matching, or master data management (MDM).
**Candidates must complete a 45-minute online assessment covering Databricks, Python, and PySpark.**
skills:
Spark SQL,Spark,Apache Spark,automated tests,automated testing,Azure Data Factory,Azure data,batch processing,code reviews,CI/CD,data quality frameworks,ingestion pipelines,Delta Lake,data pipelines,data quality checks,Databricks,Databricks Certified Data Engineer Associate,Proficiency with Git,Great Expectations,HL7,data engineering,Azure Storage,SQL Server,production code,PySpark,Python,Proficiency in SQL,stored procedure,stored procedures,Proficient in SQL,T-SQL,Strong written communication skills,architecture,intraday,healthcare,HIPAA,life-sciences,claims,maintenance,MDM,triage,Workflows,project documentation
Equal Opportunity Employer: Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other legally protected group status.
At Randstad Digital, we welcome people of all abilities and want to ensure that our hiring and interview process meets the needs of all applicants. If you require a reasonable accommodation to make your application or interview experience a great one, please contact
Pay offered to a successful candidate will be based on several factors including the candidate's education, work experience, work location, specific job duties, certifications, etc. In addition, Randstad Digital offers a comprehensive benefits package, including: medical, prescription, dental, vision, AD&D, and life insurance offerings, short-term disability, and a 401K plan (all benefits are based on eligibility).
This posting is open for thirty (30) days.
It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability.
![]()
