Job Description: Data Engineer
Location: Remote
Min Experience: 10+ Years
Job Summary
We are looking for an experienced Data Engineer with strong expertise in Python, AWS, and AWS Glue to design, develop, and optimize scalable data pipelines and ETL workflows. The ideal candidate should have hands-on experience with PySpark, cloud-based data processing, and data integration solutions.
Key Responsibilities
- Design, develop, and maintain data pipelines using Python, PySpark, and AWS Glue.
- Build and optimize ETL/ELT workflows for large-scale data processing.
- Develop data ingestion and transformation solutions using AWS services such as S3, Lambda, Redshift, and Athena.
- Implement workflow orchestration, monitoring, error handling, and data quality validation.
- Optimize pipeline performance, scalability, and reliability.
- Develop and maintain CI/CD pipelines for automated deployments.
- Collaborate with cross-functional teams to understand data requirements and deliver reliable data solutions.
Required Skills
- 10+ years of experience in Data Engineering, ETL development, or related roles.
- Strong hands-on programming experience in Python.
- Extensive experience with AWS Glue and AWS cloud services.
- Strong knowledge of PySpark and distributed data processing.
- Experience building and optimizing data pipelines and ETL workflows.
- Proficiency in SQL and relational databases.
- Experience with AWS S3, Lambda, Redshift, CloudWatch, and related services.
- Familiarity with Apache Airflow, CI/CD pipelines, and cloud automation.
- Strong troubleshooting, analytical, and communication skills.
Preferred Qualifications:
- Experience with AWS Step Functions, SNS, IAM, and Secrets Manager.
- Knowledge of data warehousing and data migration projects.
- Experience with data security, tokenization, and data quality practices.
