Position: AI Testing Engineer
Location: Remote
Experience: 3–4 Years Experience
Role Overview
Looking for an AI Testing Engineer to validate and assure the quality of AI/ML systems powering real-time contextual advertising on live video streams.
The role involves testing multimodal AI outputs, including video, text, audio, and transcripts, ensuring the accuracy of detected contextual moments and automating validation workflows.
This position is critical for native moments validation and system-level quality assurance in a real-time streaming environment.
Key Responsibilities:
Validate AI-generated outputs against labelled datasets and expected outcomes.
Perform content validation for native moments detection across video streams.
Design and implement automated testing frameworks using Python.
Build validation frameworks and test harnesses for:
- JSON parsing and comparison.
- Structured output validation using predefined schemas.
- Data validation and consistency checks.
Conduct multimodal validation of video, audio, and transcript outputs.
Execute system integration and end-to-end testing.
Track accuracy metrics, log defects, and perform root cause analysis. Automate regression testing and validation pipelines across datasets.
Collaborate with Applied Scientists to validate LLM and Generative AI outputs.
Work within AWS environments using services such as Amazon S3, AWS Lambda, and Amazon CloudWatch.
Required Skills & Experience:
Strong Python programming and scripting skills for automation and testing.
Experience in AI/ML model validation and QA workflows.
Hands-on experience with JSON parsing, schema comparison, and data validation pipelines.
Experience testing multimodal AI systems, including:
o Computer Vision.
o NLP.
o Audio and speech processing.
o LLM outputs and Generative AI pipelines.
- Good understanding of structured output validation and AI accuracy evaluation techniques.
Familiarity with AWS cloud services, particularly S3, Lambda, and CloudWatch.
Experience with media and video content analysis workflows is preferred. - Ability to work across both manual and automated testing approaches.
Thanks,
IT First Source
