Overview
Skills
Job Details
The AWS Data Engineer will be responsible for designing, building, and testing robust and scalable data pipelines and data products within the Quality Data Lake environment. This role requires expertise in AWS native services, data modeling, ETL processes, and ensuring data quality and integrity for analytical and reporting purposes.
Key Responsibilities:
Design, develop, and implement scalable data pipelines using AWS native services such as AWS Glue, Lambda, S3, Redshift, Kinesis, and others as required.
* Build and optimize data products within the Quality Data Lake, ensuring data is readily accessible and consumable for various stakeholders.
Collaborate with data scientists, analysts, and other engineering teams to understand data requirements and translate them into technical solutions.
* Data Management & Quality:
* Implement and maintain data ingestion, transformation, and loading (ETL/ELT) processes for diverse data sources into the Quality Data Lake.
* Ensure data quality, accuracy, and consistency throughout the data pipeline and within I the data lake.
* Develop and implement data governance and security best practices within the AWS environment.
* Testing & Operations:
* Design and execute comprehensive testing strategies for data pipelines and data products to ensure reliability and performance.
* Monitor, troubleshoot, and optimize data pipelines and data lake components for optimal performance and cost efficiency.
* Provide operational support and maintain documentation for all data engineering processes and solutions.
Qualifications:
* Bachelor's degree in Computer Science, Information Technology, or a related field.
* Proven experience as a Data Engineer with a strong focus on AWS cloud environments.
* In-depth knowledge and hands-on experience with AWS native services, including Glue, Lambda, S3, Redshift, Kinesis, Athena, and others relevant to data engineering.
* Strong understanding of data warehousing concepts, data modeling, and ETL/ELT proceses.
* Proficiency in SQL and experience with various database technologies.
* Familiarity with data governance, data security, and compliance principles.