Overview
Remote
$50 - $55
Contract - Independent
Contract - W2
Contract - 12 Month(s)
Skills
Amazon SQS
Amazon Web Services
Apache Kafka
Collaboration
Data Processing
Electronic Health Record (EHR)
Kubernetes
Java
Big Data
Amazon EC2
Apache Flink
Amazon S3
Apache Spark
DevOps
Orchestration
Microservices
Splunk
Streaming
Batch Processing
Job Details
Job Title: Big Data Engineer - Scala, Java
Location: Remote
Experience: 10+
About Role:
We are looking for an experienced Big Data Engineer with a strong background in Scala, Java, and a deep understanding of streaming and batch processing frameworks such as Apache Flink, Apache Spark, and Apache Kafka. The ideal candidate should have hands-on experience with AWS cloud services (EMR, Lambda, SQS, S3, EC2), containerization tools like Docker, orchestration using Kubernetes, and exposure to CI/CD pipelines with GitLab.
Key Responsibilities:
- Design, develop, and maintain scalable data processing pipelines using Apache Flink and Apache Spark.
- Work on real-time and batch data processing using Kafka and Flink.
- Write clean, efficient, and testable code in Scala and Java.
- Implement data integration solutions leveraging Kafka topics and streaming APIs.
- Deploy, manage, and monitor Big Data applications on AWS (EMR, S3, Lambda, EC2, SQS).
- Use Docker and Kubernetes (k8s) for containerization and orchestration of microservices and batch jobs.
- Set up CI/CD pipelines with GitLab for automated testing and deployment.
- Monitor and troubleshoot systems using Splunk and other observability tools.
- Ensure data quality, security, and governance across the platform.
- Collaborate with cross-functional teams including data scientists, DevOps, and product stakeholders.
Required Skills & Qualifications:
- Strong programming experience in Scala and Java.
- Hands-on experience with Apache Flink, Apache Spark, and Apache Kafka.
- Deep understanding of stream processing and event-driven architectures.
- Proficient with AWS services such as EMR, Lambda, SQS, S3, EC2.
- Working knowledge of containerization (Docker) and orchestration with Kubernetes.
- Experience in monitoring and log analysis using Splunk or equivalent tools.
- Good understanding of CI/CD processes using GitLab or similar tools.
- Strong problem-solving skills, with the ability to debug and optimize complex systems.
- Excellent communication and documentation skills.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.