Lead Data Platform Engineer

About the Role

A growing, multi-product enterprise organization in the financial data space is seeking a Lead Data Platform Engineer to architect and scale its centralized Databricks lakehouse. This is a highly strategic, infrastructure-focused role responsible for designing and implementing the data platform that powers analytics and AI across multiple product lines. The organization is actively modernizing its ecosystem, migrating multi-terabyte legacy datasets into Databricks while establishing scalable governance, performance optimization, and operational best practices.

*Unfortunately this role does not offer sponsorship. U.S. Citizens/ Holders Only*

Details:

• Full-Time, Permanent Position


• $180k - $220k + Benefits


• San Diego, CA | Hybrid - 3 Days On-Site

Key Responsibilities

• Architect and implement a centralized Databricks lakehouse


• Design unified data models across multiple product lines


• Optimize Delta Lake tables (Z-ordering, compaction, clustering) at multi-terabyte scale


• Establish Unity Catalog governance and multi-catalog access strategies


• Develop declarative ETL pipelines using Delta Live Tables


• Lead large-scale legacy data migrations into a unified platform


• Implement cost monitoring, data quality frameworks, and platform reliability standards

Required Qualifications

• 8+ years of experience in data engineering or data platform roles


• 3+ years of hands-on Databricks experience


• Strong expertise in Unity Catalog, Delta Lake optimization, and Delta Live Tables


• Proficiency in PySpark and Databricks SQL


• Experience leading a major data platform build or migration initiative


• AWS experience (S3, IAM, VPC)


• Infrastructure-as-Code experience (Terraform preferred)

Oscar Associates Limited (US) is acting as an Employment Agency in relation to this vacancy.