As part of the Big Data Engineering and Analytics organization, this role will focus on enabling a modern data lake and data mesh architecture that supports high volume, high velocity, and high variety data across enterprise and platform domains.
The Lead Data Engineer will bring deep hands-on expertise in Kubernetes, Kafka streaming platforms. This role partners closely with data architects, analytics teams, and platform engineering to deliver reliable, governed, and high-quality data solutions.
Responsibilities
- Lead the design and implementation of scalable big data pipelines for batch and real time processing
- Build and operate streaming data platforms using Kafka
- Design and deploy cloud native data solutions across AWS
- Develop and manage containerized workloads using Kubernetes
- Enable data mesh architecture with domain oriented data products
- Design and implement data lake and lakehouse architectures
- Ensure data quality, reliability, and observability
- Implement governance capabilities including metadata and lineage
- Collaborate with analytics, AI, and business teams
- Optimize performance and cost efficiency
- Mentor engineering teams
Requirements
- 10+ years of experience in big data engineering
- Strong experience with Kafka
- Strong experience with Kubernetes
- Experience with AWS
- Experience building data lakes or lakehouse platforms
- Experience with data mesh concepts
- Strong programming skills in Python, Scala, or Java
- Experience with Spark, Flink, or Beam
- Experience with Airflow or orchestration tools
- Understanding of data modeling and governance
- Experience with CICD and infrastructure automation
- Experience supporting AI and machine learning workloads including large language models such as Claude or similar platforms
- Strong communication and leadership skills
Never miss an opportunity! Create an alert based on the job you applied for.
