Role: Data Engineering POD Lead
Location: Irvine, CA
Domain: BFSI (Banking, Financial Services & Insurance)
Role Type: Data Engineering Leadership / POD Management
(Data Platform & Engineering Delivery Lead)
Job Description
Position Summary
We are seeking an experienced Data Engineering POD Lead to lead and manage a high-performing Data Engineering POD responsible for delivering scalable, reliable, and enterprise-grade data solutions. The ideal candidate will combine strong technical expertise in modern data platforms with leadership capabilities to drive end-to-end delivery, operational excellence, platform reliability, and business alignment.
This role requires hands-on experience with Databricks, Apache Spark, Airflow/Astronomer, DevOps practices, and Data Engineering frameworks, along with the ability to manage multiple teams, stakeholders, and production support activities.
Key Responsibilities
Leadership & Delivery Management
- Lead and manage the Data Engineering POD, ensuring successful execution of data initiatives.
- Own end-to-end delivery of data engineering projects and platform enhancements.
- Align POD objectives with business priorities and enterprise strategic goals.
- Drive timely project execution while ensuring SLA compliance and service quality.
- Manage cross-team dependencies, risks, escalations, and issue resolution.
- Collaborate closely with business stakeholders, program managers, and technology leadership.
Data Engineering & Platform Development
- Design, develop, and optimize scalable data pipelines and processing frameworks.
- Ensure data quality, integrity, governance, and performance across the data ecosystem.
- Implement best practices for data engineering, architecture, and operational excellence.
- Support enterprise data initiatives involving large-scale data processing and analytics.
Databricks & Apache Spark
- Lead development and administration activities on the Databricks platform.
- Optimize Spark workloads through advanced performance tuning and resource management.
- Drive platform scalability, reliability, and cost optimization initiatives.
Workflow Orchestration
- Design and manage enterprise-grade workflows using Airflow and Astronomer.
- Develop and maintain reliable DAGs to support data integration and processing requirements.
- Implement orchestration best practices to improve operational efficiency and reliability.
DevOps & Automation
- Implement CI/CD pipelines to streamline application and platform deployments.
- Utilize Infrastructure as Code (IaC) principles for environment provisioning and management.
- Drive automation initiatives to improve operational efficiency and reduce manual effort.
Platform Operations & Reliability
- Ensure platform stability, high availability, and scalability.
- Define and implement monitoring, alerting, and observability frameworks.
- Establish incident management processes and operational readiness standards.
- Oversee production support activities and continuous service improvement initiatives.
- Ensure platform reliability through proactive monitoring and performance management.
Required Qualifications
- Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field.
- 10+ years of experience in Data Engineering, Big Data, or Data Platform environments.
- 3+ years of experience leading Data Engineering teams or PODs.
- Strong hands-on experience with:
- Databricks
- Apache Spark
- Apache Airflow / Astronomer
- Data Pipeline Development
- Data Quality & Governance
- CI/CD Pipelines
- Infrastructure as Code (Terraform, CloudFormation, etc.)
- Platform Monitoring & Production Support
- Experience managing enterprise-scale data platforms and distributed teams.
- Strong stakeholder management and communication skills.
Preferred Qualifications
- Experience with AWS, Azure, or Google Cloud Platform.
- Knowledge of Lakehouse Architecture and Modern Data Platforms.
- Familiarity with Site Reliability Engineering (SRE) practices.
- Experience supporting BFSI (Banking, Financial Services, and Insurance) applications.
- Relevant Databricks, Cloud, or Data Engineering certifications preferred.
Required Skills
Databricks | Apache Spark | Airflow | Astronomer | Data Engineering | ETL/ELT | Data Pipelines | Data Quality | DevOps | CI/CD | Infrastructure as Code (IaC) | Platform Engineering | Monitoring & Alerting | Incident Management | Production Support | Cloud Platforms (AWS/Azure/Google Cloud Platform) | Team Leadership | Stakeholder Management
Never miss an opportunity! Create an alert based on the job you applied for.
