Apply Now

Sr. Databricks Engineer (PySpark and Data Lake) Exp. || 5 Days Onsite

Wilmington, DE, US • Posted 1 day ago • Updated 1 day ago

Contract Corp To Corp

Contract W2

No Travel Required

On-site

Depends on Experience

Fitment

Dice Job Match Score™

⭐ Evaluating experience...

Job Details

Skills

Databricks
ETL
AbInitio

Summary

Job Title: Sr. Databricks Engineer (PySpark and Data Lake) Exp.

Location: Wilmington, DE – 5 Days onsite role

Duration: Full Time / Long Term Project

Overview

We are seeking a Data Engineer to lead the modernization of legacy ETL systems by migrating Ab Initio workflows to scalable, modular PySpark pipelines on Databricks. The role involves transforming complex data ecosystems into cloud-native architectures while ensuring data integrity, performance, and reliability.

Key Responsibilities

ETL Modernization & Development

Analyze and migrate legacy ETL workflows from Ab Initio to PySpark-based pipelines
Design and develop scalable data pipelines on Databricks
Refactor monolithic processes into modular, reusable components
Leverage existing enterprise datasets to avoid redundancy

Data Integration & Processing

Build and maintain ETL/ELT pipelines integrating data from Snowflake and other sources
Process and publish enriched datasets for downstream applications
Support batch and near real-time data processing

Data Lineage & Optimization

Create end-to-end data lineage and data flow diagrams
Identify redundancies and drive process consolidation and optimization
Ensure adherence to data governance and quality standards

Testing & Validation

Develop unit, integration, and reconciliation frameworks
Perform dual-run comparisons with legacy systems
Validate outputs in UAT and pre-production environments

Deployment & Operations

Support cutover and migration strategy from legacy systems
Decommission legacy workflows and optimize scheduling (e.g., Control-M)
Develop runbooks, monitoring, and operational documentation

Collaboration

Work with data architects, analysts, and downstream application teams
Coordinate user acceptance testing (UAT/FAT) and stakeholder sign-offs

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 10113809
Position Id: 106509-1090-146103
Posted 1 day ago

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Data engineer / Databricks

Delaware City, Delaware

•

Today

Role: Data engineer / Databricks Experience: 10 years Location: Delaware, Wilmington (Onsite Interview) Required skills: We are seeking a Data Engineer to lead the modernization of legacy ETL systems by migrating Ab Initio workflows to scalable, modular PySpark pipelines on Databricks. The role involves transforming complex data ecosystems into cloud-native architectures while ensuring data integrity, performance, and reliability. Key Responsibilities ETL Modernization & Development Analyze a

Easy Apply

Contract, Third Party

Depends on Experience

Data Scientist - Pharma Onsite

Wilmington, Delaware

•

Today

Hiring! Data Scientist with a strong Manufacturing background for our direct client located in Wilmington, Delaware. This is an onsite job requirement. Seeking Candidates that will work on a W2 basis for 12 month + contract. Collaborate with stakeholders to identify the business requirements and the expected outcome;Identify and select relevant and credible data sources, and provide technical instructions to data analysts to collect, integrate and mine data;Identify and analyze patterns. Design

Easy Apply

Contract

Depends on Experience

Azure databricks

Wilmington, Delaware

•

26d ago

Key Responsibilities Design and implement ETL/ELT pipelines using Databricks and Apache Spark.Optimize data workflows for performance, scalability, and cost efficiency.Collaborate with data scientists, analysts, and business stakeholders to deliver data-driven solutions.Manage and monitor Databricks clusters, jobs, and workflows.Integrate Databricks with Azure/AWS/Google Cloud Platform services (depending on organization s cloud strategy).Ensure data quality, governance, and security compliance

Easy Apply

Full-time

Up to $120,000

Telecom ETL Data Architect (Contract W2)

Hybrid in West Chester, Pennsylvania

•

Today

Data Architecture & Engineering Design and architect scalable, high-performance ETL/ELT data pipelines using PySpark, Python, and Spark clusters.Develop data models and frameworks aligned with telecom processes such as buyflow, billing, usage, customer management, and order processing.Build, optimize, and monitor pipelines on Databricks (Delta Lake, Workflows, Cluster configuration).Define and enforce ETL standards, data quality rules, and engineering best practices.Domain & Business Integration

Easy Apply

Contract

Depends on Experience

Search all similar jobs