Overview
Skills
Job Details
Key Responsibilities:
Design, build, and maintain ETL/ELT pipelines for ingesting and transforming large-scale data from multiple sources.
Develop and optimize data architectures using AWS services such as S3, Glue, Redshift, Lambda, EMR, and Athena.
Implement data lake and data warehouse solutions in AWS.
Work closely with data analysts, data scientists, and business teams to ensure data availability, quality, and reliability.
Automate data workflows and manage data pipelines for performance and scalability.
Monitor, troubleshoot, and optimize data processes for cost and performance efficiency.
Ensure data security, compliance, and governance according to company standards.
Collaborate with DevOps teams for CI/CD integration and infrastructure-as-code using Terraform or CloudFormation.
Required Skills and Qualifications:
4 8 years of experience in data engineering or related roles.
Strong hands-on experience with AWS Cloud services:
S3, Glue, Redshift, Lambda, EMR, Athena, Kinesis, Step Functions.
Proficiency in Python or Scala for data processing.
Experience with SQL and database technologies (PostgreSQL, MySQL, etc.).
Experience with ETL tools and data pipeline orchestration (e.g., Apache Airflow, AWS Glue, Step Functions).
Understanding of data lake and data warehouse concepts.
Familiarity with big data frameworks (e.g., Spark, Hadoop).
Knowledge of DevOps concepts, CI/CD, and Infrastructure-as-Code (Terraform, CloudFormation).
Strong problem-solving and communication skills.