Senior Data Engineer – Databricks / PySpark / Delta Lake

Job Title: Senior Data Engineer – Databricks, PySpark, Delta Lake
Location: USA (100% Remote)
Duration: 3 Months (Initial Contract, with potential extension)
Experience Required: 10+ Years

Role Overview

We are seeking a highly experienced Senior Data Engineer who can take ownership of designing, building, and optimizing enterprise-scale data pipelines on Databricks using PySpark and Delta Lake within an Azure cloud environment. This role requires strong hands-on expertise and the ability to deliver production-quality solutions in a fast-paced, agile setting.

Key Responsibilities

• Design, develop, and maintain end-to-end data pipelines using Databricks and PySpark

• Implement ETL/ELT frameworks for large-scale batch and/or streaming data

• Build and manage Delta Lake tables, ensuring ACID compliance, schema enforcement, and schema evolution

• Apply data modeling techniques (fact/dimension models, SCDs) to support analytics and reporting

• Optimize Spark jobs for performance, scalability, and cost efficiency

• Collaborate with cross-functional teams including product owners, architects, and DevOps

• Participate in agile ceremonies and contribute to sprint planning and delivery

• Ensure production-grade code quality, including error handling, logging, testing, and documentation

• Troubleshoot and resolve data pipeline and performance issues

Required Skills & Qualifications

• 10+ years of experience in Data Engineering

• Strong, hands-on experience with:

  • Databricks

  • PySpark

  • Delta Lake

• Solid understanding of:

  • Data pipelines and ETL architecture

  • Distributed data processing and Spark internals

  • Data modeling and analytics-driven design

• Proven experience working in Azure cloud environments, including services such as:

  • Azure Data Lake Storage (ADLS Gen2)

  • Azure Data Factory (ADF)

  • Azure Synapse (preferred)

  • Azure Key Vault

• Experience delivering production-quality code using Git and CI/CD practices

• Ability to work independently and collaboratively under tight timelines

Preferred Qualifications

• Databricks Certification (Associate or Professional)

• Experience with performance tuning, job orchestration, and Delta optimization strategies

• Exposure to streaming pipelines and structured streaming is a plus