Spark Job Migration Specialist

San Francisco, CA, US • Posted 12 hours ago • Updated 12 hours ago

Contract W2

Contract Corp To Corp

On-site

Depends on Experience

Fitment

Dice Job Match Score™

🎯 Assessing qualifications...

Job Details

Skills

Amazon S3
Amazon Web Services
Apache HBase
Apache Hadoop
Apache Hive
Apache Kafka
Apache NiFi

Summary

We are looking for Spark Job Migration Specialist for our client in San Francisco, CA

Job Title: Spark Job Migration Specialist

Job Location: San Francisco, CA

Job Type: Contract

Job Overview:

Pay Range: $60hr - $65hr

Responsibilities:

Migrate JVM workloads and Spark-Submit tasks to Databricks JAR tasks or Notebook tasks.
Convert HiveQL scripts and Oozie workflows into optimized Spark SQL or PySpark applications.
Refactor data pipelines from Azure Synapse to cloud platforms, updating dependencies and notebook references.
Implement performance optimization techniques including Adaptive Query Execution (AQE) in Spark 3.
Perform regression testing to validate data consistency between legacy and new systems.
Develop validation scripts to ensure output accuracy and reliability.
Customize Spark jobs using spark context configurations for better monitoring and troubleshooting.
Reconfigure job properties, cluster settings, and Spark configurations for optimized execution.
Ensure schema evolution, data correctness, and validation using golden datasets.

Required Skills And Qualifications:

5+ years of experience with Apache Spark (PySpark or Scala).
Strong experience with Hadoop ecosystem including HDFS, Hive, HBase, and MapReduce.
Experience in migrating data pipelines to cloud or enterprise data platforms.
Proficiency in SQL and performance tuning techniques.
Hands-on experience with distributed data processing frameworks.
Experience with scripting languages such as Python, Shell, or Scala.
Familiarity with data ingestion tools like Sqoop, Kafka, and NiFi.
Experience working with cloud storage solutions such as ADLS, S3, or Blob Storage.

Preferred Skills And Qualifications:

Experience with Databricks platform and notebook-based development.
Familiarity with Azure and AWS cloud environments.
Strong understanding of data pipeline architecture and optimization strategies.
Experience handling large-scale data migrations and transformations.
Knowledge of cluster tuning and Spark configuration best practices.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 10516350
Position Id: CA_SJMP_0323
Posted 12 hours ago

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Spark job migration specialist

San Francisco, California

•

Today

Position: Spark job migration specialist SFO, CA A Spark job migration specialist migrates data pipelines, JAR tasks, and analytics workloads from legacy systems (like Hadoop/CDH or AWS EMR) to ACOS modern platforms This involves refactoring code (e.g., Hive to PySpark), performance testing, and updating Spark 2.x to 3.x. Key Job Responsibilities Workload Migration: Migrate JVM workloads and Spark-Submit tasks to Databricks JAR tasks or Notebook tasks.Pipeline Re-engineering: Convert existing Hi

Easy Apply

Contract

Depends on Experience

Big Data Migration Engineer

San Francisco, California

•

Today

Job Title: Senior Big Data Engineer (Apache Spark, Databricks, Cloud Migration) Location : Location: Onsite (SFO, CA) / Remote OK Job Description:We are looking for a Senior Big Data Engineer with strong expertise in Apache Spark, Databricks, and Cloud Migration (AWS/Azure). The ideal candidate will be responsible for migrating legacy data pipelines and Spark workloads to modern cloud-based platforms. Key Responsibilities: Migrate Spark jobs, JAR tasks, and data pipelines from legacy systems (Ha

Easy Apply

Contract

Depends on Experience

Resident Solution Architect Databricks & Spark

Remote

•

Today

Job Title:Databricks Resident Solutions Architect (RSA) Experience:12+ Years(with min. 5+ years deep specialization in Databricks/Spark) Location:Remote Role Overview We are looking for an experienced Resident Solution Architect with strong expertise in the Databricks Lakehouse Platform to design, architect, and deliver scalable, secure, and production-grade Data & AI solutions. The ideal candidate should have deep hands-on technical expertise, strong consulting experience, and proven experience

Easy Apply

Contract

Depends on Experience

Sr. Data Architect

Elk Grove, California

•

Today

Location: Elk Grove, CA Description: Hybrid role 3X a week in office in Elk Grove, CA; no remote capabilities. This is a direct hire opportunity. Summary: We're looking for a Senior Data Architect to lead the design and optimization of our Databricks Lakehouse platform on Google Cloud Platform. In this role, you'll guide a team of data engineers, own end-to-end data architecture, and build scalable pipelines that support analytics, reporting, and machine learning across the organization. Th

Contract

Search all similar jobs