Digital Annotation & Model Evaluation Specialist

Digital Annotation & Model Evaluation Specialist

Overview

We are seeking a detail-oriented and analytical Digital Annotation & Model Evaluation Specialist to support the development and improvement of AI/ML models. In this role, you will annotate digital content, evaluate model outputs, and provide structured feedback to enhance the accuracy, fairness, and performance of AI systems.

Responsibilities

• Annotation & Labeling
• Annotate datasets (text, image, audio, or video) according to defined guidelines.
• Ensure annotations are consistent, high quality, and completed within deadlines.
• Identify edge cases and ambiguities in annotation tasks and escalate for clarification.
• Only addition is minimum 20 hrs per week commitment
• Model Evaluation
• Assess AI/ML model outputs against gold-standard annotations.
• Evaluate model performance across different scenarios (accuracy, relevance, bias, safety).
• Provide structured qualitative and quantitative feedback on system outputs.
• Quality Assurance
• Conduct audits of peer annotations to ensure consistency.
• Suggest improvements to annotation guidelines and workflows.
• Collaborate with data scientists, product managers, and engineers to refine evaluation criteria.

Qualifications

• Nice to have but not mandatory: Bachelor s degree in Linguistics, Computer Science, Data Science, Cognitive Science, or related field (or equivalent work experience).
• Strong attention to detail and ability to follow complex instructions.
• Experience with annotation platforms (e.g., Labelbox, Prodigy, Appen, Scale AI, or similar).
• Familiarity with AI/ML concepts, NLP, computer vision, or generative models a plus.
• Strong written and verbal communication skills.
• Ability to work independently as well as collaboratively in a distributed/remote environment.

Preferred Skills

• Prior experience in data labeling, content moderation, or AI model evaluation.
• Knowledge of data quality metrics and model performance measures (precision, recall, F1 score, etc.).
• Multilingual skills are a plus for language annotation tasks.
• Understanding of ethical AI practices, bias detection, and fairness considerations.

Location & Work Environment

• Remote or hybrid, depending on team needs.

Flexible hours with expectations around availability for team syncs.