Apply Now

Data Engineer - Hadoop to Databricks Migration

Hybrid in Chantilly, VA, US • Posted 5 hours ago • Updated 5 hours ago

Contract W2

On-site

Fitment

Dice Job Match Score™

🧠 Analyzing your skills...

Job Details

Skills

databricks
Git
Performance Tuning
Big Data
Apache Spark
Data Migration
Problem Solving
Data Pipelines
workflows
Data Mapping
PySpark
Microsoft Azure
Data Validation
Decommissioning
Amazon Web Services
Information Engineering
Continuous Integration
Communication Skills
Extract Transform Load (ETL)
Python (Programming Language)
Data Systems
SQL Databases
Apache Hive
Data Lakes
Software Deployment
Apache Hadoop
Software Troubleshooting
Standard Sql
Hadoop Distributed File System
Cloudera
MapReduce
Migration Testing
System Migration

Summary

Senior Data Engineer Hadoop to Databricks Migration

Location: Remote
Employment Type: W2

Job Description

We are looking for an experienced Senior Data Engineer with strong hands-on experience in Hadoop to Databricks migration.

The candidate will be responsible for migrating legacy Hadoop, HDFS, Hive, and Spark workloads to Databricks and modernizing existing data pipelines using PySpark and Delta Lake.

Key Responsibilities

Migrate Hadoop and HDFS workloads to Databricks.
Analyze existing Hadoop, Hive, and Spark environments.
Migrate large-scale datasets and Hive tables to Databricks.
Convert legacy Hive, MapReduce, and Spark jobs to PySpark.
Build and maintain ETL/ELT pipelines using Databricks.
Develop data solutions using Python, PySpark, Spark SQL, and SQL.
Implement Delta Lake and Databricks Lakehouse solutions.
Perform source-to-target data mapping and transformation.
Validate migrated data for accuracy and completeness.
Troubleshoot data migration and pipeline issues.
Optimize Spark and PySpark jobs for performance.
Support migration testing, production deployment, and post-migration validation.
Work with architects and engineering teams during the migration process.
Document migration processes and technical solutions.

Required Skills

8+ years of Data Engineering or Big Data experience.
Strong Hadoop to Databricks migration experience.
Strong experience with Hadoop, HDFS, Hive, and Spark.
Hands-on experience with Databricks.
Strong Python, PySpark, Spark SQL, and SQL skills.
Experience converting legacy Hadoop workloads to Databricks.
Strong ETL/ELT pipeline development experience.
Experience with Delta Lake and Databricks Lakehouse.
Experience with data validation and reconciliation.
Strong Spark performance tuning experience.
Experience with Git and CI/CD.
Strong troubleshooting and communication skills.

Preferred Skills

Cloudera or Hortonworks to Databricks migration experience.
Unity Catalog.
Databricks Workflows and Jobs.
Auto Loader and Delta Live Tables.
AWS, Azure, or Google Cloud Platform experience.
Hadoop modernization and decommissioning experience.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 91134888
Position Id: 2026-3044/22308
Posted 5 hours ago

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Sr. Databricks Consultant

Fairfax, Virginia

•

Today

Job Description ECS is seeking a Sr. Databricks Consultant to work remotely . Please Note: This position is contingent upon additional funding. We are seeking a skilled individual with deep experience in Databricks to join our team. The primary responsibility of this role is to consult with customers to design, optimize, and fine-tune their Databricks environments for advanced data analytics, machine learning, and AI-driven insights. Leveraging tools such as Delta Lake, MLflow, and Databrick

Full-time

USD 175,000.00 - 250,000.00 per year

Direct Hire - Databrick Governance Engineer - Reston, VA

Reston, Virginia

•

4d ago

Who we are? At Hexaware Technologies, we are a leading global IT Services company, dedicated to driving digital transformation and innovation for businesses around the world. Founded in 1990, Hexaware has grown into a global trusted partner for enterprises, offering comprehensive AI empowered services including IT Consulting, Application Development, Infrastructure and Cloud Management and Business Process services. At Hexaware we are a community of creative, diverse, and open-minded Hexawarian

Easy Apply

Full-time

$80,000 - $120,000

Databricks Solutions Architect

Arlington, Virginia

•

Today

Job#: 3035708 Job Description: Databricks Solutions Architect - QA Location: Online Role Overview We are seeking a skilled DoD SECRET Cleared Databricks Solutions Architect to join a professional services team. This position involves working with customers on short- to medium-term engagements, providing technical guidance, architecture, and hands-on support to ensure they maximize value from their Databricks environments. You will collaborate with clients to design, implement, and optimize d

Easy Apply

Full-time

$190,000 - $210,000

Data Engineer (Senior Consultant)

Remote or McLean, Virginia

•

Today

Job Family: Data Engineering & Architecture Consulting Travel Required: Up to 10% Clearance Required: Ability to Obtain Public Trust What You Will Do: Guidehouse is seeking a Data Engineer (Senior Consultant) to support a Centers for Medicare & Medicaid Services (CMS) data modernization initiative. This role will focus on building and maintaining modern, scalable data pipelines and lakehouse architectures that enable secure, compliant, and high-performance processing of federal health data.

Full-time

USD 85,000.00 - 141,000.00 per year

Search all similar jobs

Remote jobs at Interon IT Solutions LLC