Overview
On Site
Hybrid
$85 - $100 hr
Contract - W2
Contract - Independent
Skills
DATA ENGINEER
ENGINEER
DATABRICKS
DATA BRICKS
HEALTH CARE
HEALTHCARE
Job Details
JOB DESCRIPTION Senior Databricks Data Engineer
(Hybrid Newark, NJ 2 days/month onsite, with flexibility to come in as needed)
Contract to Hire
We re hiring a Senior Databricks Data Engineer to help build Horizon s next-generation healthcare data platform. You ll design large-scale pipelines, implement medallion architecture, optimize Spark performance, and shape the modeling patterns for analytics, clinical, and enterprise data.
If you're a true builder who knows the difference between "running Spark jobs" and actually engineering data systems, this is your playground.
Senior Databricks Data Engineer Responsibilities
(Hybrid Newark, NJ 2 days/month onsite, with flexibility to come in as needed)
Contract to Hire
We re hiring a Senior Databricks Data Engineer to help build Horizon s next-generation healthcare data platform. You ll design large-scale pipelines, implement medallion architecture, optimize Spark performance, and shape the modeling patterns for analytics, clinical, and enterprise data.
If you're a true builder who knows the difference between "running Spark jobs" and actually engineering data systems, this is your playground.
Senior Databricks Data Engineer Responsibilities
- Architect, design, and develop large-scale ETL/ELT pipelines in Databricks, Spark, and Delta Lake.
- Implement medallion architecture and best practices for scalable data modeling.
- Build workflows, notebooks, jobs, and orchestration patterns within Databricks.
- Optimize Spark performance (clusters, caching, partitions, Z-ordering, autoscaling, shuffles).
- Apply data quality frameworks (expectations, validations, profiling).
- Work with cross-functional partners to translate requirements into actionable data solutions.
- Contribute to governance: lineage, cataloging, RBAC, compliance, metadata management.
- Troubleshoot and resolve issues across ETL pipelines, notebooks, and compute clusters.
- Mentor junior engineers as the team expands.
- Expert-level experience with Databricks, Spark, PySpark, and Delta Lake.
- Strong SQL and Python.
- Proven experience designing complex pipelines and distributed jobs.
- Experience with cloud platforms (Azure or AWS).
- Solid understanding of data modeling (conceptual, logical, physical).
- Experience with healthcare datasets (claims, eligibility, provider, clinical) is a major plus.
- Ability to work end-to-end: design build optimize document.
- Knowledge of FHIR or other healthcare interoperability standards.
- Experience with orchestration tools (Airflow, ADF, Databricks Jobs).
- CI/CD for data engineering.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.