System One
System One

AI Evaluation Specialist

Hybrid in Atlanta, GA, US • Posted 10 hours ago • Updated 10 hours ago
Full Time
Hybrid
$155000/yr
Company Branding Image
Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Job Details

Skills

Summary

Job Title: AI Evaluation Specialist 
ship or required per government contract
Clearance: Must be able to obtain Public Trust
Salary: $155K

Job Description:
In this position, you’ll evaluate enterprise AI tools and capabilities to determine their effectiveness, safety, usability, and readiness for business adoption. You’ll partner with product managers, data scientists, AI engineers, software engineers, security and privacy teams, and business stakeholders to define evaluation criteria and conduct structured assessments of generative AI products and platforms, including tools such as OpenAI and life sciences models like Rosalind. You will design repeatable evaluation frameworks, test real-world use cases, analyze model and product behavior, and translate findings into clear recommendations. You will be responsible for comparing AI solutions across quality, accuracy, reliability, safety, privacy and security. 


Key Job Responsibilities

  • Plan and execute hands-on evaluations of AI tools such as OpenAI and life sciences models like Rosalind using representative business scenarios, workflows, prompts, datasets, and user personas. 
  • Assess AI solutions for response quality, factual accuracy, relevance, consistency, reasoning, instruction following, reliability, latency, usability, accessibility, and overall task effectiveness. 
  • Design and maintain standardized evaluation frameworks, scorecards, rubrics, and acceptance criteria for enterprise AI tools, models, agents, and AI-enabled applications. 
  • Evaluate safety, privacy, security, data handling, governance, and responsible AI considerations in collaboration with security, privacy, legal, and governance stakeholders. 
  • Conduct comparative evaluations and proof-of-concept assessments across multiple AI products, models, configurations, and vendors, including analysis of functionality, integration requirements, performance, and cost. 
  • Evaluate AI agents and tool-enabled workflows, including retrieval-augmented generation (RAG), enterprise search, file and data analysis, connectors, APIs, and Model Context Protocol (MCP)-enabled capabilities where applicable. 
  • Stay current with advancements in generative AI, model evaluation, AI safety, responsible AI, and emerging enterprise AI products and incorporate relevant methods into the evaluation program. 
  • Engage in knowledge sharing and contribute to building a strong, collaborative team environment. 
Job Requirements:
  • 5+ years of experience in model evaluation, model quality assurance with hands-on experience evaluating AI-enabled products or services 
  • Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria 
  • Experience with hands-on generative AI tools and large language models, including prompt design, systematic testing, output analysis, and identification of hallucinations and other failure modes 
  • Experience assessing AI systems across dimensions such as accuracy, relevance, consistency, safety, bias, privacy, security, usability, performance, and cost 
  • Ability to demonstrate familiarity with common generative AI architectures and capabilities, including large language models, retrieval-augmented generation (RAG), AI agents, APIs, enterprise search, and tool or connector integrations 
  • Bachelor's degree 
Job Desirable Skills:
  • Experience evaluating enterprise AI platforms or assistants, including OpenAI products and life sciences models like Rosalind or comparable generative AI solutions 
  • Experience with automated LLM evaluation frameworks, model-as-a-judge approaches, human-in-the-loop evaluation, red teaming, or adversarial testing 
  • Experience evaluating RAG systems, AI agents, MCP-enabled tools, enterprise connectors, or multi-step AI workflows 
  • Experience developing dashboards or reporting mechanisms for AI evaluation metrics, quality trends, risk findings, and regression results 
  • Ability to work with datasets and evaluation results using tools such as Python, SQL, spreadsheets, notebooks, or analytics platforms to calculate metrics, identify patterns, and summarize findings 
  • Possession of an industry-recognized certification in AI, machine learning, data science, cybersecurity, cloud technologies, or responsible AI 
  • Master’s degree preferred 

#M1
#LI-VH1
Ref: #851-Rockville-S1
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: 10295162
  • Position Id: 382967
  • Posted 10 hours ago

Company Info

About System One

System One is a leading provider of specialized, highly technical outsourced services, workforce solutions, staffing, and recruiting to critical infrastructure, technology, life sciences, and government sectors As an essential partner to private and public organizations of various needs and sizes, we offer our expertise to help them bring their most complex and mission-critical programs to fruition.

With our highly specialized services, in-demand technical skills, and vast operational expertise, we can deliver accelerated results that benefit our clients and the job seekers who partner with us. We staff contract, contract-to-hire, and direct-hire professionals across our network to work with our clients, some of whom are industry leaders and nationally branded organizations.

Our dedicated team of recruiters work tirelessly to match jobseekers with careers that align with their skills, aspirations, and goals.

We understand that finding the right job can be a transformative experience, and we are committed to supporting jobseekers throughout their career journeys. Our recruitment services are designed to connect individuals with fulfilling jobs that offer growth and development opportunities.

Whether you're an organization seeking customized workforce solutions, outsourced services, or if you're a jobseeker searching for your next opportunity,

System One is the perfect staffing partner for you.

About_Company_One
Go to company profile
Contact the job poster
VH

Viktoria Hotvianska

Recruiter @ System One

View Profile
Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.
More jobs at System One in Atlanta, GA