Looking for AI Prompt & Data Quality Specialist

Job Title: AI Prompt & Data Quality Specialist

Location: US Remote

Duration: 6+ months

We are looking for a meticulous AI Prompt & Data Quality Specialist to lead our "Human-in-the-Loop" workflows. In this hybrid role, you will not only design the prompts that drive our AI, but you will also define the Ground Truth for what "good" looks like. You will be responsible for creating high-quality evaluation datasets, grading model outputs, and refining instructions to ensure our AI models are accurate, safe, and aligned with user needs.

1. Prompt Engineering & Tuning

• Draft and develop complex system prompt for various use cases (e.g., summarization, extraction, persona-based chat).
• Implement advanced strategies (Chain-of-Thought, Few-Shot) to guide the model through complex reasoning tasks.
• Iterate on prompt versions to reduce latency and token usage while maintaining quality.

2. Data Labeling & Quality Assurance (New)

• Review and Annotate: Systematically review model outputs to tag them for accuracy, tone, safety, and hallucinations.
• RLHF Support: Perform "A/B ranking" on model responses (choosing the better of two options) to support Reinforcement Learning from Human Feedback.
• Create Golden Sets: Write "ideal" human-generated responses to serve as Ground Truth for automated testing and fine-tuning.
• Maintain Guidelines: Develop and update the Annotation Rubric the detailed instructions that define exactly what counts as a "correct" or "safe" response.

Must-Haves

• Linguistic Precision: Exceptional command of English grammar and semantics. You can spot the difference between "good" and "great" writing instantly.
• Experience with LLMs: Familiarity with GPT-4, Claude, or similar models, including their limitations (hallucinations, bias).
• Extreme Attention to Detail: Ability to perform repetitive data review tasks without losing focus or consistency.