AI/LLM Quality Assurance Lead

Role Overview

We are seeking an experienced QA / Validation Engineer to support the testing and validation of Large Language Models (LLMs) within a large financial institution. This role will focus on ensuring model accuracy, reliability, compliance, and risk alignment when applied to banking use cases such as customer interactions, fraud detection, and operational support.

You will work at the intersection of AI, data quality, and regulatory compliance—helping ensure LLM outputs are safe, explainable, and production-ready.

Key Responsibilities

• Design and execute test strategies for LLM-based applications (chatbots, summarization, document processing, etc.)
• Develop and maintain LLM evaluation frameworks, including:
  • Prompt testing and variation analysis
  • Response accuracy and relevance validation
  • Hallucination detection and mitigation
• Create and execute test cases using real-world banking scenarios (e.g., payments, fraud alerts, account inquiries)
• Validate outputs against regulatory and compliance requirements (e.g., consumer protection, data privacy)
• Perform bias, fairness, and toxicity testing on model outputs
• Conduct data validation and lineage checks for training and inference datasets
• Partner with Data Scientists, Risk, Compliance, and Engineering teams to refine model behavior
• Build automated test suites and integrate into CI/CD pipelines
• Track, document, and report defects, risks, and model performance metrics

Required Qualifications

• 5+ years of QA / Testing experience, preferably in banking or financial services
• Hands-on experience testing AI/ML or LLM-based systems
• Strong understanding of:
  • NLP concepts (tokenization, embeddings, prompt engineering)
  • Model evaluation metrics (precision/recall, BLEU, ROUGE, etc.)
• Experience designing test frameworks for non-deterministic systems
• Proficiency in at least one programming language (Python preferred)
• Experience with test automation tools and frameworks
• Strong SQL and data validation skills
• Familiarity with model risk management (MRM) and regulatory expectations in banking

Preferred Qualifications

• Experience with LLM platforms (e.g., OpenAI, open-source models like LLaMA)
• Knowledge of Responsible AI principles (fairness, explainability, governance)
• Experience testing chatbots or contact center AI solutions
• Familiarity with tools such as:
  • Prompt management/evaluation tools
  • Data quality tools (e.g., Great Expectations)
• Prior exposure to fraud, AML, payments, or customer servicing domains

Key Skills

• LLM Testing & Validation
• Prompt Engineering & Evaluation
• Data Quality & Analysis
• Automation & Scripting (Python)
• Risk & Compliance Awareness
• Critical Thinking & Problem Solving

About Techstra Solutions   
Techstra Solutions is a certified woman-owned (WBENC) management consulting firm specializing in strategy, technology, and implementation services for large organizations undergoing digital and talent transformation. Our experienced team partners with clients to co-create innovative solutions in applications, data, AI, and automation that accelerate measurable, sustainable change. From advisory consulting through technical execution, we are dedicated to driving world-class business solutions that fit your strategic requirements and deliver results. For more information: