AI Evaluation & Data Engineering Specialist- 7+ yrs- New York, NY- Onsite

Job description:

 

 AI Evaluation & Data Engineering Specialist Role Overview We are looking for AI Evaluation & Data Engineering Specialists to design, curate, and operationalize datasets and evaluation frameworks for AI product performance assessment. This role involves working with large language models (LLMs), human raters, and automation tools to measure model accuracy, correctness, and usability. Key Responsibilities Build and maintain evaluation datasets for AI models across programming languages (Python, Golang, JavaScript, Java). Develop and apply data labeling and scoring guidelines based on Google’s evaluation framework. Implement LLM-judge calibration workflows to align automated and human evaluations. Perform error analysis, drift detection, and regression testing of AI model outputs. Collaborate with automation engineers to integrate datasets into evaluation pipelines. Support rater training, inter-rater reliability checks, and dataset validation reviews. Manage data quality assurance and documentation for contributions to Google-maintained repositories. Required Skills & Experience 4–8 years of experience in AI/ML data operations, evaluation, or data engineering. Proficiency in Python (mandatory) for dataset manipulation, analysis, and scripting. Experience with LLM evaluation, prompt engineering, or text generation quality assessment. Familiarity with Gemini CLI, Vertex AI, or LangChain evaluation tools. Strong understanding of data curation, annotation workflows, and labeling quality metrics. Hands-on with Git-based repositories and CI/CD data workflows. Excellent analytical and problem-solving skills with attention to detail. Preferred Qualifications Experience evaluating code-generation or NLP-based AI products. Exposure to data governance and privacy compliance frameworks. Background in computer science, data science, or linguistics preferred.