Data Engineer- Pyspark Scala

Overview

On Site
$100,000 - $120,000
Full Time
100% Travel

Skills

Pyspark
Scala
SQL
Snow Flake Schema

Job Details

Job Title : Data Engineer- Pyspark Scala

Location : Irving,Texas ( Onsite )

Job Type : Full-time

Required Qualifications:

At least 6 years of Information Technology experience.

Experience in Pyspark & Scala

Experience in Kafka streaming.

Solid understanding and hands-on experience in SQL, Unix shell scripting

Preferred Qualifications:

Proficiency in programming languages such as Scala, Java, or Python

At least 4 years of experience in Apache Spark, including Spark Core, Spark SQL, Spark Streaming, and Spark MLlib

Lead the development and optimization of batch and real-time data pipelines, ensuring scalability, reliability, and performance.

Architect, design, and deploy data integration, streaming, and analytics solutions leveraging Spark, Kafka, and Snowflake.

Extensive hands-on experience with Apache Kafka, including topics, partitions, producers, consumers, and stream processing

Solid understanding of distributed computing concepts, data partitioning, fault tolerance, and parallel processing

Experience with cluster management tools like Apache Mesos or Apache YARN is desirable

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.