Position: AI SDET/Evaluation Engineer
Location: New York City – Hybrid
Employment Type: Full Time
Note: This Position Is for an AI SDET / Evaluation Engineer. This Person Will Be Using AI Coding Agents to Do the Coding SDET Engineers Used to Do. This Is A QA Position Testing Applications and The Company Own AI Systems Using AI Coding Agents Set Up by The Candidate.
Job Description:
Build and run tests that assess the quality and reliability of the firm’s AI applications. The Evaluation Engineer helps maintain evaluation datasets, investigates failures and provides clear evidence of how AI systems perform on legal and business tasks, working with senior engineers, legal specialists and application QA.
Key Responsibilities
· Automated evaluations: Implement automated evaluation cases and supporting scripts using established frameworks and guidance from senior engineers.
· Test datasets: Prepare and maintain test datasets, source documents and reference examples with legal specialists, following requirements for confidentiality, permitted data use and version control.
· Quality assessment: Run evaluations of answer accuracy, completeness, citations, retrieval relevance, instruction following and appropriate escalation. Apply agreed scoring criteria and flag cases requiring expert judgement.
· Workflow testing: Execute functional and integration tests for AI workflows, including document ingestion, tool calls, structured outputs and error handling, in coordination with application QA.
· Failure analysis: Review outputs and execution traces to identify failures. Reproduce issues, document supporting evidence and help distinguish data, retrieval, model and application defects.
· Version comparison: Compare results across model, prompt and application versions. Report changes accurately, including uncertainty and limitations, rather than relying solely on headline scores.
· Grader calibration: Support the calibration of automated graders by comparing their scores with human reviews and investigating inconsistencies.
· Regression suites: Maintain regression suites, contribute to CI/CD integration and turn approved production failure examples into new test cases.
Required Qualifications
· Minimum five years’ experience developing software, demonstrated through employment, placements or substantial projects, using programming languages such as Python, Java or C#.
· Foundational conceptual understanding of large language models, including their capabilities, limitations and common failure modes, and the ability to apply that understanding when developing evaluation criteria, test cases and acceptance thresholds with guidance from senior engineers.
· Ability to use coding agents effectively to support development, testing and analysis, with awareness of best practices for prompting, context management, validation, security, responsible use of generated code and when to seek human review.
· Working knowledge of APIs, Git, automated testing, software development lifecycle practices and CI/CD tools such as Jira and Jenkins.
· Foundational understanding of statistics and measurement, including sampling, precision and recall.
· Strong analytical skills, attention to detail and the ability to document issues clearly and seek technical or legal guidance when needed.
· Demonstrated curiosity and ability to learn in a rapidly evolving AI landscape, assess new models, tools and evaluation techniques, and apply relevant learning pragmatically with appropriate guidance.
Preferred Qualifications
· Practical experience evaluating AI applications. Given how new the field is, this is not required, and relevant academic, internship or project experience will also be considered.
