Our goal is to solve problems and deliver results for our clients. At Cloud and Things, you can be a part of transforming the public sector’s IT environment. Our team is on the forefront of helping to solve the government's most complex IT challenges. If you are seeking a role that offers the opportunity to work on rewarding projects, consider a career with Cloud and Things.
*This is an exempt position. Salary commensurate with experience*
Job Title: AI Tester
Location: Albany, NY (Remote Available)
Duration: 12 Months (with potential for extension)
W2 Hourly Rate: $47-$57/hour
Application Deadline: November 8, 2026
Overview:
We are seeking an AI Tester who will support our NYS client on the Medicaid Eligibility and Client Management (MECM) modernization program. The consultant will test AI and generative AI solutions, including conversational AI, retrieval-augmented generation (RAG), and agentic AI, to ensure alignment with business, functional, and technical requirements. This role will assess output quality, validate safeguards, and provide testing metrics and readiness recommendations.
Duties:
• Develop and execute test plans, test cases, and evaluation scenarios for conversational AI, RAG, agentic AI, and other GenAI solutions.
• Assess AI outputs for accuracy, relevance, consistency, completeness, and alignment with functional, business, and technical requirements.
• Perform negative, edge-case, and adversarial testing to identify unexpected AI behavior and failure modes.
• Validate content retention, guardrails, and overall solution quality; document findings and coordinate defect resolution.
• Track test results and quality metrics and communicate clear readiness recommendations to project stakeholders.
Mandatory Qualifications:
• 2-5 years of hands-on experience as an AI Tester, including testing and quality assurance of AI solutions.
Desirable Qualifications:
• Experience with software quality assurance and testing; AI/ML or generative AI testing experience preferred.
• Familiarity with large language models (LLMs) and AI evaluation techniques.
• Experience with test management and defect-tracking tools such as Jira.
• Strong analytical skills and ability to recognize unexpected AI behavior.
• Experience working in regulated or public-sector environments.
What We Offer:
Opportunities to lead impactful transformations in the public sector infrastructure.
Engagement with high-profile government initiatives, supporting meaningful change.
Professional development through exposure to innovative Agile practices and leadership collaboration.
A collaborative environment that values ownership, initiative, and continuous learning.
Ready to make a difference?
We’re eager to connect with qualified candidates committed to delivering results and fostering excellence within client projects.
Cloud and Things complies with all applicable federal, state, and local laws regarding recruitment and hiring. All qualified applicants are considered for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, protected veteran status, or any other category protected by applicable federal, state, or local laws.
AI-Assisted Resume Evaluation Notice
Cloud and Things – Talent Management
Notice to Candidates
Cloud and Things utilizes artificial intelligence (AI) tools to assist our recruiting team in evaluating candidate applications for streamlining; consistency, efficiency, and thoroughness. All hiring decisions are ultimately made by our human recruiting professionals.
How AI Is Used
Our AI tools assist by:
Analyzing resumes against job requirements
Supporting our recruiters in candidate data evaluation
Ensuring consistent review standards across all applications
Important: AI serves as a support tool only. As noted above, all candidate selection and hiring decisions are made by experienced human recruiters. Your unedited resume will be processed by our AI tools as part of this evaluation.
Your Data and Privacy
Cloud and Things Data Handling:
Your information is processed securely and used exclusively for recruitment purposes
Cloud and Things may store your resume in our Applicant Tracking System (ATS) indefinitely for future job matching opportunities
You may opt out of long-term ATS storage by emailing your name and your request to opt out of storing your resume in the ATS to:
All personal information is handled confidentially in accordance with our privacy policy
AI Tool Data Processing:
AI processing data is retained for a maximum of 90 days, after which it is deleted
All data sent to AI tools is encrypted in transit and at rest
AI tools comply with applicable privacy laws including GDPR and CCPA
Personal data is anonymized or minimized wherever possible during AI processing
Your Participation
By submitting your application, you acknowledge this notice and consent to AI-assisted evaluation as part of our recruitment process.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: 91116955
- Position Id: 9537
- Posted 7 hours ago
