AI Data Quality Specialist position
Remote
Our Ideal Candidate
BA or BS degree +1-3 years of support and/or customer service or content design QA experience Preferred: financial institution or other highly regulated environment preferred
Ability to evaluate conversations from a variety of angles and identify specific areas of improvement
Ability to spend remain focused for long-running, repetitive tasks (eg labeling many datasets consistently)
Ability to collaborate with AI scientist, product managers, and developers to understand project requirements and objectives.
Able to evaluate a member’s experience by ensuring that we are providing the correct responses to our members with an empathetic and unbiased approach
Design and create a diverse range of evaluators (LLM and rules-based), rubrics, and prompts tailored to specific AI tasks and applications.
Develop creative and innovative ways to generate effective prompts that improve model performance.
Conduct extensive research to identify and curate relevant data sources for rubric, prompt, and evaluator development.
Ensure the judges, rubrics, and prompts are designed to elicit desired responses and achieve the desired outcomes.
Optimize and fine-tune judges, rubrics, and prompts based on continuous testing and feedback from AI models, Agent outcomes, AI Scientists, Product Managers, developers.
Analyze AI model outputs and make data-driven adjustments to improve prompt efficacy.
Stay up-to-date with the latest advancements in natural language processing and AI technologies.
Effective communicator in both verbal and written communication
Excellent organization, detail oriented, and high level of accuracy
Proven experience in resolving complex customer issues and managing escalations with internal and external teams
Analytical skills with experience using Microsoft Excel or Google Sheets, SQL, python
Applying data filters, creating charts and graphs, generating pivot tables, simple statistical analyses
Experience with CRM and QA platforms
Thanks & regards,
Kalyan Kumar
Innovative Consulting Solutions, LLC
1901 N. Roselle Rd., Suite 800
Schaumburg, IL 60195
Email id:
