InfiCare Technologies
InfiCare Technologies

Sr. Databricks Engineer (PySpark and Data Lake) Exp. || 5 Days Onsite

Wilmington, DE, US • Posted 2 days ago • Updated 2 days ago
Contract Corp To Corp
Contract W2
No Travel Required
On-site
$130 - $140/hr
Fitment

Dice Job Match Score™

🫥 Flibbertigibetting...

Job Details

Skills

  • Databrick
  • PySpark
  • Data Lake

Summary

Job Title: Sr. Databricks Engineer (PySpark and Data Lake) Exp.

Location: Wilmington, DE – 5 Days onsite role

Duration: Full Time / Long Term Project

Overview

We are seeking a Data Engineer to lead the modernization of legacy ETL systems by migrating Ab Initio workflows to scalable, modular PySpark pipelines on Databricks. The role involves transforming complex data ecosystems into cloud-native architectures while ensuring data integrity, performance, and reliability.

Key Responsibilities

ETL Modernization & Development

  • Analyze and migrate legacy ETL workflows from Ab Initio to PySpark-based pipelines
  • Design and develop scalable data pipelines on Databricks
  • Refactor monolithic processes into modular, reusable components
  • Leverage existing enterprise datasets to avoid redundancy

Data Integration & Processing

  • Build and maintain ETL/ELT pipelines integrating data from Snowflake and other sources
  • Process and publish enriched datasets for downstream applications
  • Support batch and near real-time data processing

Data Lineage & Optimization

  • Create end-to-end data lineage and data flow diagrams
  • Identify redundancies and drive process consolidation and optimization
  • Ensure adherence to data governance and quality standards

Testing & Validation

  • Develop unit, integration, and reconciliation frameworks
  • Perform dual-run comparisons with legacy systems
  • Validate outputs in UAT and pre-production environments

Deployment & Operations

  • Support cutover and migration strategy from legacy systems
  • Decommission legacy workflows and optimize scheduling (e.g., Control-M)
  • Develop runbooks, monitoring, and operational documentation

Collaboration

  • Work with data architects, analysts, and downstream application teams
  • Coordinate user acceptance testing (UAT/FAT) and stakeholder sign-offs
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: 10113809
  • Position Id: 106509-1090-
  • Posted 2 days ago
Create job alert
Set job alertNever miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Sr. Databricks Engineer (PySpark and Data Lake) Experience

Wilmington, Delaware

•

Today

Easy Apply

Contract

$60 - $65

Databricks Engineer Seeking Ex-JPMC Candidates for In-Person Interview - C2C/W2/Full-Time

Wilmington, Delaware

•

Today

Easy Apply

Contract

Depends on Experience

Data Engineer

Remote or Hybrid in Delaware City, Delaware

•

4d ago

Easy Apply

Contract

$60,000 - $65,000

Azure databricks

Wilmington, Delaware

•

21d ago

Easy Apply

Full-time

Up to $120,000

Search all similar jobs