Data Pipeline/ Migration Engineer

Overview

On Site
$70 - $80
Accepts corp to corp applications
Contract - W2
Contract - Independent
Contract - 12 Month(s)

Skills

Python
Java
SQL
PostgreSQL
Oracle
AWS
AWS S3
AWS Lambda
AWS Step Functions
AWS CloudWatch
Jupyter Notebooks
Apache Spark
Databricks
Delta Lake
Data Migration
Data Ingestion
Data Pipelines
Data Cleansing
Infrastructure as Code
IAC
Terraform
AWS CloudFormation
DevOps
CI/CD
Serverless Architecture
Microservices
Git

Job Details

Role: Data Pipeline/Migration Engineer

Location: Princeton, NJ (Onsite from day 1)

Client: Major Financial Services Company

Duration: one year

JD:

Project Description:
Migration of legacy data ingestion pipelines to a shared data management system (BDMS). The work will involve:

  • Datasets will be loaded into BDMD
  • New tools will be created to allow data analysts to modify the datasets
  • Analysts will validate and cleanse the data which will then be delivered to AWS S3
  • Data will then be enriched using technologies like Databricks, AWS Lambda & Step Functions and Jupyter Notebooks/Workflows and loaded in Delta tables.

We ll trust you to:
- Work directly with product owners, engineers to build solutions

- Work with data analysts/scientists to collect insight on possible data collection efficiencies and identify automation potential for manual workflows
- Develop infrastructure as code (IaC) and embrace DevOps principles

- Work with AWS cloud services to implement micro services using serverless technologies

 

You'll need to have:

- 4+ years of experience programming in Python or Java

- Solid understanding of database systems and SQL including Oracle and PostgreSQL

- A Degree in Computer Science, Engineering, Mathematics, similar field of study or equivalent work experience.

- AWS experience including services like S3, Lambda, CloudWatch

 

We'd love to see:

- Experience with data migrations from legacy to cloud.

- Knowledge of Spark, Databricks and IaC(Terraform/CloudFormation)

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.