LLM QA Engineer
Contract W2
12 Months
No Travel Required
Remote
$55 - $60/hr
Ohm Systems, Inc
Fitment
Dice Job Match Score™
🔗 Matching skills to job...
Job Details
Skills
- AI Testing
- QA Automation
- Generative AI
- GenAI
- Large Language Models (LLMs)
- LLM Evaluation
- RAG (Retrieval-Augmented Generation)
- Prompt Engineering
- Conversational AI Testing
- Playwright
- Selenium
- API Testing
- REST APIs
- Postman
- Python
- SQL
- Git
- CI/CD
- Regression Testing
- End-to-End (E2E) Testing
- Functional Testing
- AI Safety Testing
- Human-in-the-Loop (HITL)
- AI Observability
- OpenAI
- Azure OpenAI
- ChatGPT
- Claude
- LangChain
- LangGraph
- CrewAI
- Model Context Protocol (MCP)
- Pinecone
- ChromaDB
- Weaviate
- RAGAS
- DeepEval
- Docker
- Kubernetes
- GitHub Actions
- JavaScript
- Agile/Scrum
- Artificial Intelligence
- Healthcare
Summary
Job Description
- We build AI-powered solutions including conversational AI, RAG systems, AI agents, and enterprise automation platforms.
Role
- Join the AI Engineering team to test and optimize Generative AI, conversational AI, and RAG applications. Focus on AI quality, benchmarking, safety, observability, and automation to ensure production-ready systems.
Responsibilities
- Test Generative AI and conversational systems
- Perform LLM evaluation and benchmark testing
- Validate RAG (retrieval quality, relevance, grounding)
- Build automation frameworks (Playwright/Selenium)
- Conduct API, regression, and E2E testing
- Perform AI safety, red teaming, and bias validation
- Support observability, monitoring, and HITL workflows
- Collaborate with engineering, QA, and DevOps teams
- Support CI/CD quality pipelines
Required Skills
- QA & automation testing
- Playwright/Selenium, API testing (Postman, REST)
- GenAI & RAG validation, LLM evaluation
- Prompt engineering, conversational AI testing
- HITL testing, AI safety & observability
- SQL, Git, Python, CI/CD concepts
Preferred
- OpenAI, Azure OpenAI, ChatGPT, Claude
- LangChain, CrewAI, LangGraph, MCP
- Vector DBs (Pinecone, ChromaDB, Weaviate)
- RAGAS, DeepEval
- Python/JS, Docker, Kubernetes, GitHub Actions
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: 10111826
- Position Id: 9013182
- Posted 3 hours ago
Company Info
Ohm Systems, Inc. specializes in IT and Healthcare staffing services, dedicated to linking highly skilled professionals with our public and private clients across the United States. Our track record showcases our commitment to delivering outstanding staffing and consultancy solutions to our clients. We prioritize diversity and inclusivity and take pride in being an employer that promotes equal opportunities and affirmative action. Our goal is to foster an inclusive work environment that embraces individuals from all backgrounds, irrespective of their gender, race, or orientation.
Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.Similar Jobs
It looks like there aren't any Similar Jobs for this job yet.
Search all similar jobs