Intermediate Java Developer (Databricks)

Task Description

The Intermediate Java Developer will be responsible for converting existing PySpark codebases into optimized Java-based Spark applications. This role includes developing, refactoring, and maintaining scalable data processing solutions developed on the Databricks platform (or similar Spark execution environments).

The developer will collaborate with IRS stakeholders and technical teams to ensure accurate translation of logic, high performance of Spark jobs, and compliance with federal security, privacy, and governance standards.

Key Responsibilities

• Convert existing PySpark applications into equivalent, efficient Java Spark implementations

• Design, develop, and maintain scalable Spark-based data pipelines
• Implement data processing logic using Java 8+ with best practices in OOP and functional programming
• Integrate solutions with IRS datasets including IRMF, BMF, and IMF
• Optimize Spark jobs for performance, maintainability, and cost-efficiency
• Collaborate across development, data engineering, and architecture teams
• Troubleshoot and debug Spark workloads in distributed environments
• Ensure compliance with IRS data handling, security, and governance policies

Required Skills & Qualifications

Security Requirements

• Active IRS MBI Clearance required

• IRS-issued laptop strongly preferred
• Candidate must provide a copy of their active MBI clearance letter

Education

• Bachelor’s degree in computer science, Information Systems, or related field

Technical Experience (5+ Years Total Professional Experience)

IRS Data Systems

• Hands-on experience working with IRMF, BMF, or IMF datasets

• Understanding of IRS data structures, formats, and security protocols

Programming Languages

• Strong expertise in Java 8+

• Functional programming experience (Streams API, Lambdas)
• Solid understanding of object-oriented design patterns

Apache Spark

• Proficiency with Spark Core, Spark SQL, and DataFrame/Dataset APIs

• Familiarity with RDD concepts and appropriate use cases
• Experience with PySpark-to-Java Spark translation preferred
• Performance tuning and Spark job optimization
• Ability to use Spark UI for debugging and troubleshooting

Big Data Ecosystem

• Familiarity with HDFS, Hive, or HBase

• Experience with S3, Kafka
• Knowledge of Parquet, Avro, ORC formats

Data Processing / ETL

• Strong understanding of batch and streaming data processing

• Experience building and supporting ETL pipelines
• Data cleansing, transformation, enrichment skills

DevOps / Deployment

• Experience with Databricks, or EMR

• CI/CD experience (Jenkins, GitHub Actions)
• Monitoring tools such as Grafana, Prometheus, Datadog, or Spark logs preferred

Version Control & Build Tools

• Proficient with Git

• Experience with Maven or Gradle

Testing

• Unit testing with JUnit or TestNG

• Experience with Mockito or similar frameworks preferred
• Data validation and regression testing for Spark workloads
• Develop unit tests for Spark transformations, actions, and business logic using frameworks such as Junit
• Validate data correctness, schema evolution, null handling, boundary conditions, and exception scenarios
• Validate input/output schemas, partitioning strategies, and data lineage across Spark jobs
• Support testing of backward compatibility and migration scenarios during modernization
• Compare legacy vs modernized outputs to ensure functional parity and identify intentional deviations
• Analyze defects raised during system, integration, and UAT testing to determine root cause (code, data, environment, or test issue)
• Implement code fixes, data corrections, or configuration changes, and add regression tests to prevent recurrence
• Support re-testing and validation cycles, ensuring fixes are stable in distributed environments
• Work with QA team to review test strategies, test cases, and acceptance criteria early in the development lifecycle
• Provide technical guidance on Spark internals, data flows, and expected behaviors to improve test effectiveness
• Ensure tests are CI/CD-ready, automated, and meet agreed code coverage and quality gates and established operational standards

Soft Skills / Engineering Practices

• Experience working in Agile/Scrum environments

• Strong documentation abilities (Confluence, etc.)
• Ability to troubleshoot production issues efficiently
• Strong communication and collaboration skills

Preferred Qualifications

• Experience specifically converting PySpark workloads into Java Spark

• Experience with Databricks, Google Dataproc, or similar Spark ecosystems
• Knowledge of Delta Lake or Apache Iceberg
• Big data performance modeling and tuning