Annotation Data Engineer

We are seeking a detail-oriented and analytical Annotation Data Engineer to support annotation quality, machine learning data operations, and dataset validation for AI/ML initiatives. The ideal candidate will have experience in annotation quality management, data quality engineering, and quality measurement systems, preferably within physical AI, robotics, computer vision, or foundation model development.
Key Responsibilities

• Define, implement, and maintain annotation quality standards, guidelines, defect taxonomies, and acceptance criteria.
• Develop and maintain quality metrics and reporting frameworks to measure accuracy, consistency, completeness, rework rates, throughput, and reviewer agreement.
• Design and execute sampling strategies, annotation audits, escalation workflows, and root-cause analysis processes for large-scale annotation programs.
• Evaluate generated captions, reasoning outputs, and foundation model training datasets against established quality benchmarks.
• Analyze annotation quality trends, identify recurring defects, and coordinate corrective and preventive actions with cross-functional teams.
• Validate dataset releases to ensure compliance with defined quality requirements and readiness for model training.
• Maintain traceable quality records, audit documentation, issue logs, and continuous improvement plans.
• Conduct calibration sessions and support reviewer alignment to improve annotation consistency and inter-annotator agreement.
• Collaborate with annotation operations, data engineering, quality engineering, and AI/ML teams to optimize annotation workflows.
• Leverage SQL, Python, and reporting tools to analyze quality data, automate measurements, and generate actionable insights.

Required Technical and Functional Skills

• 2-5 years of relevant experience in annotation quality, machine learning data operations, quality engineering, or data quality management.
• Experience supporting physical AI, robotics, computer vision, or machine learning programs.
• Demonstrated experience developing quality metrics, KPIs, and performance measurement systems.
• Hands-on experience working with large-scale annotation programs and dataset quality assurance processes.
• Experience evaluating generated captions, reasoning outputs, or foundation model training datasets.
• Strong understanding of annotation guidelines, sampling plans, acceptance criteria, defect classification, and audit methodologies.
• Ability to perform quality trend analysis, identify root causes, and drive corrective actions.
• Working knowledge of SQL and Python for data analysis, quality measurement, and reporting.
• Experience creating quality dashboards, reports, and performance tracking mechanisms.
• Strong analytical, problem-solving, documentation, and cross-functional communication skills.