AWS Data Engineer

Responsibilities:

• Design, build, and maintain scalable data ingestion, transformation, and delivery pipelines using AWS services such as Glue, Lambda, ECS, S3, Amazon MSK, AWS Transfer Family, AWS SNS and SQS, AWS EventBridge, EMR, Kinesis, Redshift, RDS, DynamoDB.
• Design and implement scalable ETL/ELT frameworks and data ingestion pipelines to process structured and semi-structured data across cloud-based data platforms. 
• Implement CI/CD automation for data pipelines using GitHub and GitHub Actions.
• Set up SFTP data ingestion pipelines and install/configure SSH to enable secure authentication, file movement, and system connectivity.
• Monitor data pipeline performance using Datadog, integrated with CloudWatch metrics and logs for unified visibility and automated alerts.
• Design and maintain scalable data pipelines leveraging IceKafKa for real time event streaming
• Re-engineer legacy ETL logic into modern, event driven or micro batch pipelines.
• Build and optimize data interfaces, APIs, and connectors to deliver curated datasets into the SaaS platform.
• Work with Fivetran ELT platform or similar CDC tools.
• Align data models, validation rules, and ingestion formats with SaaS vendor specifications.
• Implement automated data quality checks, reconciliation rules, and validation frameworks to ensure accuracy and completeness of migrated data.
• Conduct parallel runs and data comparisons between legacy outputs and cloud/SaaS outputs.
• Provision and manage AWS infrastructure resources using Infrastructure as Code (IaC) tools such as Terraform and AWS CloudFormation.
• Apply AWS best practices for IAM, encryption (KMS), VPC networking, and secure data transfer.

Requirements:

• Strong experience with AWS data services (S3, Glue, Redshift, RDS, DynamoDB, Lambda).
• Proficiency in Python and SQL for data transformation and pipeline development.
• Experience setting up and managing SFTP based data ingestion, including secure file transfers, directory structures, and automated ingestion workflows.
• Ability to install, configure, and manage SSH for secure access, authentication, and data movement across systems.
• Hands-on experience implementing CI/CD automation for data pipelines using GitHub and GitHub Actions.
• Experience configuring monitoring and observability using Datadog, including dashboards, metrics, logs, and alerting.
• Working knowledge of AWS CloudWatch, including metrics, log groups, alarms, and integrations with external monitoring tools.
• Experience with Infrastructure as Code (Terraform or CloudFormation).
• Ability to troubleshoot pipeline failures, performance issues, and monitoring alerts across distributed data systems.
• Familiarity with cloud based data engineering environments and modern DevOps practices.

Education:

• Bachelor's degree in Computer Science/Engineering or a related field