AWS Data Engineer

AWS Data Engineer

Remote with Contract

Job Description

Responsibilities: -

1. AWS (Amazon Web Services):

• Core Services: Proficiency with core AWS services like EC2, S3, RDS, Lambda, and VPC.
• Data Services: Experience with AWS data services Glue and EMR.
• AWS DMS: Knowledge of AWS Database Migration Service (DMS) for migrating databases to AWS.
• CDC: Understanding of Change Data Capture (CDC) techniques to capture and replicate changes from source databases to target databases.
• Security: Understanding of AWS security best practices, IAM, and encryption.

1. Databricks:

• PySpark & Spark SQL: Strong analytical skills in PySpark & Spark SQL for big data processing and analysis.
• Delta Live Tables: Expertise in using Delta Live Tables for building reliable and scalable data pipelines.
• Notebooks: Strong utilization of Databricks Notebooks for data analysis.
• Workflows : Setting up and monitoring Databricks Workflows.
• Data Integration: Experience integrating Databricks with AWS services.

1. DevOps Principles:

• CI/CD Pipelines: CI/CD pipelines using Azure Pipelines.
• Version Control: Proficiency with Azure Repos and Git for version control.
• Automation: Scripting and automation using PowerShell, Bash, or Python. Automating the build, test, and deployment processes

1. Infrastructure as Code (IaC):

• Terraform: Experience with Terraform for managing AWS and Azure infrastructure.

1. On Prem integration with AWS

• Integrating on prem data with AWS and Databricks.
• Thoroughly test and validate the data to ensure it has been transferred correctly and is fully functional.

1. Optimization and Monitoring:

• Optimize AWS services and Databricks for performance and cost-efficiency.
• Proficiency in setting up monitoring and logging using tools like AWS CloudWatch to track the performance and health of the complete data flow.