Job Details

About the Role:

We are initiating a large-scale evaluation project that involves assessing and annotating coding prompts across 10 programming languages. The work includes reviewing model-generated responses, validating code correctness, evaluating logic, and ensuring adherence to required technical standards. This role requires strong coding knowledge, analytical skills, and high attention to detail.

Priority Distribution Guidance:

High-priority languages account for 80% of annotations: Python, Java, JavaScript, C/C++, HTML/CSS, Bash/Shell. Among these, Python, Java, JavaScript, and C/C++ should contribute equally to the 80% split. Medium-priority languages (20%): TypeScript, Rust, PHP, SQL.

Preferred Qualifications:

Education: Technical degree preferred
Experience: Minimum 3+ years of relevant experience
Skills:
- Strong coding expertise in the required tech stack
- SDE-level technical understanding
- Excellent analytical skills and attention to detail
Tools: SRT/Gala Tool

Responsibilities:

Assess and annotate coding prompts in various programming languages
Review model-generated responses and validate code correctness
Evaluate logic and ensure adherence to required technical standards
Contribute to the distribution of annotations based on language priorities
Provide feedback and recommendations for improvement

Requirements:

Required Skills: Bash, C, C++, Java, JavaScript, Python, Rust, SQL, TypeScript, PHP

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.