Lead AWS Data Engineer

Introduction:

This position is for a Lead AWS Data Engineer who will be responsible for designing, developing, and maintaining ETL/ELT pipelines using AWS services such as Glue, Lambda, and Step Functions. The role involves implementing data ingestion frameworks, ensuring data quality and security, building data lakes and warehouses, and collaborating with various stakeholders to deliver reliable datasets.

Responsibilities:

1. Design, develop, and maintain batch and streaming ETL/ELT pipelines using AWS services (Glue, Lambda, Step Functions, etc.).
2. Implement data ingestion frameworks from diverse sources (APIs, databases, streaming platforms).
3. Ensure data quality, governance, and security across all pipelines.
4. Build and optimize data lakes and warehouses leveraging Amazon S3, Redshift, Athena, and Lake Formation.
5. Collaborate with data scientists, analysts, and business stakeholders to deliver reliable datasets.
6. Monitor and troubleshoot data workflows, ensuring high availability and performance.
7. Stay updated with emerging AWS technologies and recommend improvements.

Requirements:

Required Skills & Qualifications:

1. Strong experience with AWS cloud services: S3, Glue, Redshift, EMR, Kinesis, Lambda, DynamoDB, RDS, Athena, SNS, SQS.
2. Proficiency in SQL, Python, and PySpark.
3. Knowledge of data modeling, schema design, and performance tuning.
4. Familiarity with CI/CD pipelines and DevOps practices tool (Terraform, CloudFormation).
5. Experience with security best practices (IAM, encryption, compliance).