AWS Data Engineer

  • Torrance, CA
  • Posted 2 days ago | Updated 2 days ago

Overview

Hybrid
Up to $150,000
Full Time

Skills

AWS
Glue
Python
ETL
Redshift
Lambda
S3
Apache

Job Details

Mandatory Skills : Apache Spark, Java, Athena, Aurora, Python, RDS, Redshift, AWS Glue, AWS Lambda, AWS S3,Scala,ANSI-SQL,Aws Step Functions, Python for DATA
Good to Have Skills : Apache Spark, Python, RDS, Redshift, AWS Glue, AWS Lambda, AWS S3, Scala, ANSI-SQL, Aws Step Functions

Job description:

5 years of experience in data engineering database design ETL processes and data warehousing.

3 years of experience with AWS tools and technologies S3 EMR Glue Athena RedShift RDS Spectrum and Airflow.

2 years of experience with CICD tools.

Strong knowledge of data storage and processing technologies including databases and data lakes based distributed computing frameworks eg Hadoop Spark.

3 in programming languages such as Python Java or Scala.

Nice to have Informatica Cloud experience IDMC.

Detailed Responsibilities

Develop and Maintain Data Integration Solutions

Design and implement data integration workflows using AWS Glue,EMRAWSMWAA Airflow Lambda Redshift

Demonstrate proficiency in Pyspark Apache Spark and Python for data processing large datasets

Ensure data is accurately and efficiently extracted transformed and loaded into target systems

Ensure Data Quality and Integrity

Validate and cleanse data to maintain high data quality

Ensure data quality and integrity by implementing monitoring validation and error handling mechanisms within data pipelines

Optimize Data Integration Processes

Enhance the performance optimization of data workflows to meet SLAs scalability of data integration processes and cost efficiency on AWS cloud infrastructure

Identify and resolve performance bottlenecks finetuning queries and optimizing data processing to enhance Redshifts performance

Regularly review and refine integration processes to improve efficiency

Support Business Intelligence and Analytics

Translate business requirements to technical specifications and coded data pipelines

Ensure timely availability of integrated data for business intelligence and analytics

Collaborate with data analysts and business stakeholders to meet their data requirements

Maintain Documentation and Compliance

Document all data integration processes workflows and technical system specifications

Ensure compliance with data governance policies industry standards and regulatory requirements

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.