Sr Databricks Developer

• 8+ years experience in data engineering with at least 5 years of Databricks hands-on experience in AWS environments.
• Deep expertise in PySpark, Spark (on Databricks), AWS cloud data services (S3, Glue, Redshift, Lambda).
• Strong foundation in SQL, ETL/ELT pipeline development, and data warehousing concepts.
• Proven experience working with healthcare data sets (claims data highly preferred) and understanding of U.S. regulatory standards (e.g., HIPAA).
• Hands-on with code migration, data integration, and optimizing big data workloads in Databricks.
• Familiarity with data quality, lineage, and governance tools in AWS (Glue Data Catalog, Lake Formation, etc.).
• Excellent stakeholder management, business communication, and team collaboration skills

Job Description:

• Design, build, and maintain high-performance data pipelines and ETL workflows in Databricks on AWS, processing both healthcare and claims data.
• Develop and optimize Spark applications using PySpark and/or Scala on Databricks Unified Analytics Platform.
• Migrate and integrate data from various sources (including PostgreSQL or other RDBMS), leveraging AWS native services (S3, EMR, Glue, Redshift).
• Ensure HIPAA compliance and security standards throughout data engineering processes.
• Perform data modeling, transformation, and quality checks on large healthcare datasets (claims, EHR, EMR, payer-provider).
• Collaborate with multi-disciplinary teams to deliver business-focused healthcare analytics solutions.
• Implement CI/CD pipelines and DevOps best practices for reliable and automated deployments.
• Mentor junior engineers and establish data engineering best practices