Data Engineer

Introduction:

This position is for a Data Engineer who will be responsible for building a comprehensive QA framework in Azure DataBricks. The individual will work on gathering, storing, detecting, and processing large volumes of data for QA analysis and bug detection in ETL code.

Responsibilities:

• Develop QA ETL pipelines
• Ensure data quality in the QA framework
• Optimize data storage solutions for efficient processing
• Collaborate with QA analysts to test code and data as part of the SDLC process

Requirements:

Required Skills:

• Python programming language proficiency
• Strong SQL knowledge
• Experience with cloud data tools such as AWS Glue, Azure Data Factory, and Azure DataBricks

Preferred Skills:

• Previous experience in Quality Assurance (QA) processes
• Familiarity with CyberDW and SMBC Data Lakehouse
• Understanding of medallion architecture for ETL processes

The Security Data Operations team is seeking a qualified candidate who can design and develop a robust QA framework and pipeline for the CyberDW on the SMBC Data Lakehouse. The role will involve analyzing and developing ingestion pipelines to test data feeds from raw to target processes. The individual will serve as a technical liaison between IT and CDAO Data Engineering teams, ensuring seamless communication and collaboration.

The selected candidate will also be responsible for creating JIRA tickets for developers to address bugs in their code promptly. The QA processes developed must be flexible, adaptable, and scalable to meet evolving business needs. Timely follow-ups and updates to the CyberDW sprint schedule, as well as adherence to internal IT SDLC agile processes, are crucial components of this role.