Data Engineer

Senior Data Engineer
Remote 

Visa Status:  / / Any Independent Visa.

Must Have 10+ Years of Experience 

Domain / Industry: IT (Healthcare)

Top Required Skills

• Databricks (Workspace, Jobs, Clusters, Performance Tuning)
• PySpark (Advanced DataFrame & Spark SQL development)
• Delta Lake (ACID transactions, MERGE, Schema Evolution)
• Data Engineering & ETL Pipeline Design
• Azure Cloud (ADLS Gen2, ADF, Synapse, Key Vault)
• Data Modeling (Fact/Dimension, SCD Type 1 & 2)
• Agile / Scrum methodology
• Production-grade coding, CI/CD, Git version control

Job Description (JD)

We are seeking a Senior Data Engineer with strong hands-on expertise in Databricks, PySpark, and Delta Lake to design, build, and maintain scalable, enterprise-level data pipelines in an Azure cloud environment. The ideal candidate will have deep experience delivering production-quality data solutions and working in fast-paced agile teams.

Key Responsibilities:

• Design, develop, and optimize end-to-end data pipelines using Databricks and PySpark
• Implement robust ETL/ELT frameworks for batch and streaming data processing
• Create and manage Delta Lake tables, ensuring ACID compliance, schema enforcement, and schema evolution
• Apply strong data modeling techniques to support analytics, reporting, and downstream consumption
• Optimize Spark jobs for performance, scalability, and cost efficiency
• Handle data quality, validation, error handling, and pipeline monitoring
• Collaborate with cross-functional teams including product owners, architects, QA, and DevOps
• Participate in agile ceremonies (stand-ups, sprint planning, retrospectives)
• Deliver production-ready code with proper logging, testing, documentation, and version control
• Troubleshoot and resolve data pipeline, performance, and scalability issues

Preferred Qualifications:

• Databricks Certification (Associate or Professional)
• Experience with Structured Streaming
• Prior experience supporting healthcare or retail data platforms