Lead Apache Spark Developer

Overview

Remote
$30 - $35
Full Time

Skills

Data Modeling
Apache Hadoop
Apache Hive
Apache Spark
Big Data
Collaboration
Integration Testing
Java
Optimization
Performance Tuning
Python
SQL
Data Processing
Data Quality
Data Structure
Distributed Computing
Git
HDFS
Scala
Streaming
Unit Testing
Version Control
Workflow

Job Details

Role: Lead Apache Spark Developer

Location: Remote

Duration: long term

Contract: C2C

Required experience: 10+ years

Job Summary:

We are looking for a skilled Apache Spark Developer to join our Big Data team. The ideal candidate will have hands-on experience in building scalable and efficient data processing pipelines using Spark and related big data technologies.

Key Responsibilities:

  • Design, develop, and optimize large-scale data processing pipelines using Apache Spark.
  • Work with big data platforms such as Hadoop, Hive, and HDFS.
  • Write high-performance code in Scala, Java, or Python for batch and streaming data processing.
  • Collaborate with data engineers, analysts, and architects to understand business requirements and deliver technical solutions.
  • Implement data quality checks, error handling, and logging in Spark jobs.
  • Optimize Spark jobs for performance, resource usage, and fault tolerance.
  • Maintain and improve existing data workflows and pipelines.
  • Participate in code reviews, unit testing, and integration testing.

Required Skills:

  • Strong experience with Apache Spark (RDD, DataFrame, Spark SQL, Spark Streaming).
  • Proficiency in Scala, Java, or Python.
  • Experience with Hadoop ecosystem tools (Hive, HDFS, Yarn, etc.).
  • Good understanding of data modeling, data structures, and distributed computing.
  • Familiarity with version control systems like Git.
  • Experience with performance tuning and optimization of Spark jobs.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.