![]()
Lead Data Platform Engineer - Databricks / Delta Lake / Unity Catalog - $220-$250k DOE + Benefits - San Diego, CA & Tarrytown, NY (Hybrid)
Location: San Diego, CA & Tarrytown, NY- Hybrid (in-office 2-3 days/week)
Salary: $220k-$250k base DOE
Benefits: Comprehensive health, 401(k), generous PTO, parental leave, supplemental benefits
What's in it for you?
- Lead the architecture and build of a centralized, multi-product data platform from the ground up on Databricks.
- Solve challenging data infrastructure problems at scale-multi-TB datasets, Delta Lake optimization, and cross-product governance.
- Hands-on role enabling other Data Engineers while defining platform operations, performance, and reliability standards.
- Exposure to cutting-edge technologies: Unity Catalog, Delta Live Tables, PySpark, Databricks SQL, AWS, Terraform.
- Be part of a collaborative, growth-focused team with opportunities to influence technical strategy and career development.
The Role
As Lead Data Platform Engineer, you will:
- Design and implement unified data models across multiple disparate product lines.
- Build a multi-catalog governance strategy with Unity Catalog, enabling secure isolation and controlled cross-product data sharing.
- Optimize Delta Lake tables with Z-ordering, compaction, liquid clustering, partitioning, and retention policies.
- Create declarative ETL pipelines with Delta Live Tables, orchestrating data ingestion from internal and external sources.
- Integrate third-party data sources (ERP systems, external providers) with automated, monitored ingestion.
- Establish platform operations standards: cost monitoring, performance tuning, data quality frameworks, and self-service capabilities for engineers.
Business Challenges You'll Solve
- Lead migration of multi-terabyte datasets from legacy systems to a unified Databricks lakehouse.
- Design multi-product data architectures enabling both secure isolation and analytics sharing where appropriate.
- Build a platform that scales efficiently, controlling costs while supporting analytics and AI workloads.
- Implement monitoring, alerting, and validation to ensure reliability and high-quality data.
Required Experience
Databricks Expertise
- Unity Catalog: production experience with multi-catalog governance, metastore design, lineage tracking
- Delta Lake: Z-ordering, compaction, liquid clustering, performance tuning at multi-TB scale
- Delta Live Tables: declarative ETL pipelines, change data capture, expectations/constraints
- Databricks Workflows: job orchestration, scheduling, monitoring
- PySpark & Databricks SQL: query optimization, code review, performance tuning
Core Platform Engineering
- 6-8 years in data engineering/platform roles, with 3+ years hands-on Databricks experience
- Led at least one major platform build or migration
- AWS experience (S3, IAM, VPC), Infrastructure-as-Code (Terraform preferred)
Technical Leadership
- Architect platforms from first principles and communicate technical decisions clearly
- Strong written and verbal communication for technical and business stakeholders
Preferred (Not Required)
- Experience with financial/ERP data (NetSuite)
- Exposure to AI/ML data preparation or RAG/LLM pipelines
- Familiarity with regulated industry compliance and governance frameworks
Why This Role?
- Take ownership of mission-critical data infrastructure impacting multiple business lines.
- Work on complex, large-scale datasets in a cutting-edge data platform environment.
- Opportunity to mentor and enable a team of Data Engineers while shaping long-term architecture.
- Competitive salary, bonus, benefits, and career growth in a collaborative, hybrid environment.
Apply Now - This role is exclusive to our recruiting partner, and limited interview slots are available. Contact Bryce Reading for more information or to submit your application.
Lead Data Platform Engineer / Databricks / Delta Lake / Unity Catalog / Delta Live Tables / Data Governance / PySpark / Data Engineering / AWS / Terraform / San Diego / NYC / Python
Oscar Associates Limited (US) is acting as an Employment Agency in relation to this vacancy.
Never miss an opportunity! Create an alert based on the job you applied for.
