Databricks Data Engineer

Introduction:

The Databricks Data Engineer will be responsible for hands-on DLT/LDP, delta lake, unity catalog, DAB, spark advance performance tuning, and building modular engineering frameworks. This role will require experience with Foundry as a secondary skill, with an emphasis on specific requirements.

Responsibilities:

• Hands-on experience with DLT/LDP, delta lake, unity catalog, DAB, and spark advance performance tuning
• Solid data engineering fundamentals and proven hands-on Databricks experience
• Performance tuning and optimization, including Spark internals, join strategies, and skew handling
• Efficient use of LDP APIs (CDF, CDC)
• Implementation of data quality frameworks, including expectations, quarantine logic, and alerting
• Compliance and governance tasks such as PII/PHI masking and access controls
• Orchestrating and stitching pipelines together

Requirements:

Required Skills:

• Solid data engineering fundamentals
• Hands-on experience with Databricks
• Performance tuning & optimization (Spark internals, join strategies, skew handling)
• Efficient use of LDP APIs (CDF, CDC)
• Data quality frameworks (expectations, quarantine logic, alerting)
• Compliance & governance (PII/PHI masking, access controls)

Preferred Skills:

• Experience with Foundry
• Unity catalog knowledge
• Experience with orchestrating and stitching pipelines