Job Title: Lead AWS Cloud Data Engineer - Data Mesh
Duration: 18 Month
Client: Leading Banking Institution
Location: San Francisco, CA 94105
Description:
We are seeking an experienced AWS Cloud Data Engineer to lead the design, development, and modernization of enterprise-scale cloud data platforms. This role will focus on building modern Data Mesh architectures, developing scalable AWS-based data pipelines, implementing DevSecOps best practices, and integrating AI-powered solutions to accelerate software development and data engineering initiatives.
The ideal candidate will possess strong expertise in AWS cloud services, enterprise data architecture, distributed data platforms, governance frameworks, and modern AI technologies.
Key Responsibilities:
Cloud Data Engineering
- Design, develop, and maintain scalable ETL/ELT data pipelines using AWS Glue, EMR, Lambda, Kinesis, and Step Functions.
- Build ingestion frameworks for structured, semi-structured, streaming, and API-based data sources.
- Develop optimized data models and consumption layers for enterprise analytics.
- Implement automated data quality validation, monitoring, alerting, and anomaly detection.
- Maintain metadata management, data lineage, governance, and access control policies
Data Mesh Architecture
- Design and implement enterprise Data Mesh architecture on AWS.
- Define domain ownership, data products, federated governance, and self-service data infrastructure.
- Build reusable frameworks and accelerators for publishing, discovering, and consuming data products.
- Optimize platform performance, scalability, and cloud cost efficiency.
Modern Data Platforms
Work extensively with technologies including:
- Databricks
- Unity Catalog
- Delta Lake
- Starburst / Trino
- Collibra
- Immuta
DevSecOps & Cloud Infrastructure
- Implement CI/CD pipelines using GitHub Actions, GitLab CI, or Jenkins.
- Develop Infrastructure as Code using Terraform and CloudFormation.
- Deploy and manage containerized applications using Docker and Amazon ECS.
- Implement automated security scanning, compliance validation, disaster recovery, and AWS GovCloud security controls.
AI-Augmented Engineering
- Design and develop Agentic AI solutions and Retrieval-Augmented Generation (RAG) pipelines.
- Utilize Amazon Bedrock and enterprise LLMs for intelligent automation.
- Leverage AI-assisted development tools such as GitHub Copilot and Claude Code.
- Improve software delivery through AI-powered code generation, testing, documentation, and automation.
Technical Leadership
- Lead architecture reviews and technical design discussions.
- Mentor engineering teams on modern cloud data engineering practices.
- Provide Level 3 production support.
- Collaborate with architects, product owners, and business stakeholders to deliver scalable enterprise solutions.
- Produce technical documentation and architectural standards.
Required Qualifications:
Cloud Data Engineering & AWS (5+ Years)
- 5+ years of hands-on experience designing and implementing distributed data architectures on AWS.
- Strong expertise with AWS services including:
- Amazon S3
- AWS Glue
- AWS Lake Formation
- Amazon EMR
- Amazon Redshift
- Experience building scalable ETL/ELT pipelines and enterprise data platforms.
Data Mesh Architecture
- Hands-on experience implementing Data Mesh architecture and modern distributed data platforms.
- Strong understanding of:
- Domain-oriented data ownership
- Data as a Product
- Federated Computational Governance
- Self-service data infrastructure
- Experience designing scalable, domain-driven data solutions.
Modern Data Platform Technologies
Hands-on experience with one or more of the following enterprise data platforms:
- Databricks (Unity Catalog, Delta Lake)
- Starburst / Trino (Federated Query)
- Collibra (Data Governance & Metadata Management)
- Immuta (Dynamic Data Access Control)
- Ability to architect and integrate multi-platform data solutions across enterprise environments.
DevSecOps & Infrastructure as Code (IaC)
- Strong experience implementing DevSecOps best practices throughout the software development lifecycle.
- Hands-on experience with:
- CI/CD pipelines (GitHub Actions, GitLab CI, Jenkins)
- Containerization technologies (Docker, Amazon ECS)
- Infrastructure as Code (Terraform, AWS CloudFormation)
- Experience implementing security, compliance, and governance controls within AWS cloud environments, including AWS GovCloud.
AI-Augmented Development & Engineering
- Experience leveraging Generative AI tools to improve software development productivity through:
- Code generation
- Automated testing
- Documentation
- SDLC acceleration
- Hands-on experience designing or implementing:
- Agentic AI solutions
- Retrieval-Augmented Generation (RAG) pipelines
- Amazon Bedrock or similar Large Language Model (LLM) platforms
Leadership & Communication:
- Proven ability to translate complex business requirements into scalable technical solutions.
- Experience leading technical design discussions, architecture reviews, and engineering initiatives.
- Demonstrated ability to mentor and guide engineering teams.
- Excellent verbal and written communication skills with the ability to present complex technical concepts to both technical and non-technical stakeholders.
- Experience working across hybrid cloud and on-premises enterprise environments.
Never miss an opportunity! Create an alert based on the job you applied for.