Data Engineer

Hybrid in Jersey City, NJ, US • Posted 12 days ago • Updated 11 days ago
Full Time
No Travel Required
Hybrid
$65 - $75/hr
Fitment

Dice Job Match Score™

🛠️ Calibrating flux capacitors...

Job Details

Skills

  • azure databricks
  • QA
  • AWS Glue

Summary

Data Engineer

Introduction:

This position is for a Data Engineer who will be responsible for building a comprehensive QA framework in Azure DataBricks. The individual will work on gathering, storing, detecting, and processing large volumes of data for QA analysis and bug detection in ETL code.

Responsibilities:

  • Develop QA ETL pipelines
  • Ensure data quality in the QA framework
  • Optimize data storage solutions for efficient processing
  • Collaborate with QA analysts to test code and data as part of the SDLC process

Requirements:

Required Skills:

  • Python programming language proficiency
  • Strong SQL knowledge
  • Experience with cloud data tools such as AWS Glue, Azure Data Factory, and Azure DataBricks

Preferred Skills:

  • Previous experience in Quality Assurance (QA) processes
  • Familiarity with CyberDW and SMBC Data Lakehouse
  • Understanding of medallion architecture for ETL processes

The Security Data Operations team is seeking a qualified candidate who can design and develop a robust QA framework and pipeline for the CyberDW on the SMBC Data Lakehouse. The role will involve analyzing and developing ingestion pipelines to test data feeds from raw to target processes. The individual will serve as a technical liaison between IT and CDAO Data Engineering teams, ensuring seamless communication and collaboration.

The selected candidate will also be responsible for creating JIRA tickets for developers to address bugs in their code promptly. The QA processes developed must be flexible, adaptable, and scalable to meet evolving business needs. Timely follow-ups and updates to the CyberDW sprint schedule, as well as adherence to internal IT SDLC agile processes, are crucial components of this role.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: 10148179
  • Position Id: 8891508
  • Posted 12 days ago
Create job alert
Set job alertNever miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Hybrid in New York, New York

Yesterday

Easy Apply

Full-time

$100,000 - $110,000

Remote or New York, New York

Today

Contract

USD165 - USD170

Hybrid in New York, New York

21d ago

Easy Apply

Contract

90 - 93

New York, New York

13d ago

Easy Apply

Third Party, Contract

Depends on Experience

Search all similar jobs