Job Title: Spark Scala Engineer(Need EX-Apple)
Location: Austin, TX or Sunnyvale, CA
Duration: Contract
Job Overview:
Develop and maintain high-performance Apache Spark applications using Scala.
Design and implement scalable ETL/ELT data pipelines for large and complex datasets.
Work with Spark SQL, DataFrames, Datasets, and RDDs for data processing and transformation.
Optimize Spark jobs for performance, memory utilization, and scalability.
Develop data ingestion and processing workflows from various sources such as databases, APIs, files, and streaming platforms.
Work with Kafka and other messaging/streaming technologies for real-time data processing.
Implement data quality, validation, error handling, and monitoring within data pipelines.
Work with cloud platforms such as AWS, Azure, or Google Cloud Platform and their respective data services.
Integrate Spark pipelines with Hive, HDFS, Delta Lake, Snowflake, or other data platforms.
Troubleshoot production issues and perform root-cause analysis for data processing failures.
Collaborate with data engineers, architects, analysts, and business teams to understand data requirements.
Participate in code reviews, unit testing, deployment, and CI/CD activities.