Role: Data Engineering Tower Lead
Location: Irvine, CA
(Enterprise Data Platform & Operations Lead)
Job Description
Position Summary
We are seeking an experienced Data Engineering Tower Lead to lead and manage the enterprise data engineering organization, ensuring the successful delivery, reliability, scalability, and operational excellence of the data platform. This role will be responsible for end-to-end data engineering delivery, platform performance, team leadership, production support, and alignment with business objectives.
The ideal candidate will possess strong expertise in Databricks, Apache Spark, Airflow/Astronomer, Data Engineering, DevOps, and Platform Operations, along with proven leadership experience managing multiple teams and large-scale data programs.
Key Responsibilities
Leadership & Delivery Management
- Lead and manage the entire Data Engineering Tower across multiple teams, pods, and programs.
- Own end-to-end delivery of data engineering initiatives, ensuring alignment with business priorities and enterprise goals.
- Drive program execution, on-time delivery, SLA adherence, and operational excellence.
- Manage cross-functional dependencies, risks, issues, and escalations.
- Collaborate closely with business stakeholders, product owners, and technology leadership teams.
Data Engineering & Platform Architecture
- Oversee the design, development, and optimization of scalable data pipelines and data processing frameworks.
- Ensure high standards of data quality, governance, scalability, and performance.
- Establish best practices for enterprise data engineering and platform operations.
- Lead platform modernization and continuous improvement initiatives.
Databricks & Spark Leadership
- Provide technical leadership for Databricks platform administration and optimization.
- Drive advanced Spark development, performance tuning, and workload optimization.
- Ensure efficient utilization of compute resources and platform scalability.
Workflow Orchestration & Automation
- Lead enterprise orchestration strategies using Airflow/Astronomer.
- Design and govern DAG development standards, reliability practices, and workflow optimization.
- Improve operational efficiency through automation and orchestration frameworks.
DevOps & Infrastructure Automation
- Implement and oversee DevOps best practices including CI/CD pipelines.
- Drive Infrastructure-as-Code (IaC) adoption and automation initiatives.
- Ensure streamlined deployment processes and platform consistency across environments.
Platform Operations & Reliability
- Establish and maintain highly available, reliable, and scalable data platforms.
- Define monitoring, alerting, observability, and incident management processes.
- Lead production support activities and operational readiness programs.
- Ensure platform stability, disaster recovery preparedness, and business continuity.
Required Qualifications
- Bachelor''s degree in Computer Science, Information Technology, Engineering, or a related field.
- 12+ years of experience in Data Engineering, Data Platforms, or Big Data technologies.
- 5+ years of experience leading large-scale Data Engineering teams and programs.
- Strong hands-on experience with:
- Databricks
- Apache Spark
- Apache Airflow / Astronomer
- Enterprise Data Engineering and ETL/ELT frameworks
- CI/CD pipelines and DevOps practices
- Infrastructure as Code (Terraform, CloudFormation, etc.)
- Experience managing enterprise-scale production environments and platform operations.
- Strong understanding of monitoring, observability, incident management, and reliability engineering.
- Excellent stakeholder management, communication, and leadership skills.
Preferred Qualifications
- Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform.
- Knowledge of Data Governance, Data Quality, and Data Security frameworks.
- Experience implementing enterprise-scale Data Lakehouse architectures.
- Familiarity with SRE (Site Reliability Engineering) practices and platform engineering concepts.
- Relevant cloud, Databricks, or data engineering certifications are highly desirable.
Key Skills
Databricks | Apache Spark | Airflow | Astronomer | Data Engineering | ETL/ELT | Data Pipelines | DevOps | CI/CD | Infrastructure as Code (IaC) | Platform Engineering | Monitoring & Alerting | Incident Management | Production Support | Cloud Platforms (AWS/Azure/Google Cloud Platform) | Leadership & Stakeholder Management
Never miss an opportunity! Create an alert based on the job you applied for.
