Senior Data Engineer Job Description
Role Summary
Senior engineering role responsible for designing and developing scalable AWS Databricks data pipelines,
transformation frameworks, and reusable components for complex healthcare data workloads.
Roles & Responsibilities
Design and develop PySpark and Spark SQL pipelines on AWS Databricks.
Build reusable transformation frameworks, Delta tables, workflows, and metadata/configuration-driven
processing components.
Implement complex source-to-target mappings, state-specific transformation logic, crosswalks, reference-data
processing, and business rules.
Develop batch pipelines for full-refresh, incremental, historical, and reprocessing scenarios.
Implement logging, error handling, audit controls, balance/control checks, lineage, and operational monitoring.
Optimize Spark workloads for performance, scalability, and cost.
Migrate or re-engineer legacy ETL logic from DataStage, SSIS, Informatica, C/Linux, SQL, and other batch platforms.
Support data validation, reconciliation, defect analysis, and production-readiness activities.
Participate in code reviews, technical design reviews, testing, and CI/CD.
Required Skills & Experience
10+ years of data engineering experience with strong hands-on development skills.
Strong AWS Databricks, PySpark, Spark SQL, Delta Lake, and Databricks Workflows experience.
Strong understanding of AWS S3-based data-lake patterns and secure integration with enterprise source
systems.
Strong SQL and experience building high-volume batch ETL / ELT pipelines.
Experience with reusable and metadata-driven transformation frameworks.
Experience with schema evolution, partitioning, Delta optimization, performance tuning, and large-volume data
processing.
Experience with Oracle, SQL Server, flat files, APIs, and enterprise data integrations.
Experience with Git, CI/CD, automated testing, and deployment practices.
Exposure to DataStage, SSIS, Informatica, C/Linux, or other legacy ETL technologies is preferred.
Experience with Unity Catalog, access controls, lineage, and data-governance concepts is preferred.
Domain Knowledge
Healthcare payer / Medicaid / MMIS experience is preferred.
T-MSIS or other healthcare regulatory-reporting experience is a strong plus.
Understanding of claims, encounters, eligibility, provider, managed care, pharmacy, TPL, and financial
transaction data is preferred.
Familiarity with source-to-target mappings, crosswalks, regulatory validation rules, and reconciliation is
beneficial.
