W2-Hybrid :: AI (LLM & RAG) Test Engineer with Financial/Investment Domain Exp. || G.C / U.S.C

AI (LLM & RAG) Test Engineer with Financial/Investment Domain Exp. || G.C / U.S.C

6+Months

Hybrid

#### Overview:
Join our advanced AI Quality Engineering team working at the intersection of large language models and critical enterprise technology. We deliver innovative testing and quality assurance for next-generation AI-driven systems used by premier financial services and regulated-industry clients. Your expertise will directly impact the reliability, compliance, and performance of AI solutions across complex business workflows.

#### Key Responsibilities

- **Develop & Implement AI Quality Assurance:**  
  Craft comprehensive test plans for AI/ML-powered features (LLM assistants, knowledge retrieval, workflow AI), spanning functional, end-to-end, regression, UAT, and non-functional domains.
- **Execute Thorough Evaluations:**  
  Assess system accuracy, relevance, safety, response time, and operational integrity across realistic use cases and production scenarios.
- **Construct Evaluation Assets:**  
  Create and evolve test cases, datasets, and benchmarks specific to prompt engineering, retrieval-augmented generation, and user interactions.
- **Collaborative Definition of Success:**  
  Work with development, product, and business teams to define acceptance criteria, quality gates, and KPIs for AI solutions.
- **Risk & Defect Management:**  
  Identify weaknesses (hallucinations, model drift, edge cases, workflow vulnerabilities), drive proactive risk mitigation, and document corrective actions.
- **Automate for Scale:**  
  Develop and support automation pipelines for continuous testing, model monitoring, reporting, and CI/CD integration.
- **Data & API Validation:**  
  Validate interconnected data flows, APIs, microservices, and business logic dependencies supporting AI-empowered business systems.
- **Governance & Quality Reporting:**  
  Drive release readiness, maintain risk logs, test documentation, and foster best practices for AI quality and model lifecycle management.
- **Operational Feedback & Improvement:**  
  Track user feedback, incident metrics, and production signals to guide iterative improvement.
- **Institutionalize Best Practices:**  
  Champion standards, toolkits, and controls for scalable, trustworthy AI delivery in regulated environments.

#### Requirements

- **Education:**  
  Bachelor’s degree or higher in Computer Science, Software Engineering, Data Science, Information Systems, or equivalent technical discipline.
- **Experience:**  
  - 6+ years in software quality engineering, test automation, or related technical QA roles.
  - Demonstrated experience in testing enterprise business applications, APIs, ETL/data pipelines, or workflow platforms.
  - Hands-on evaluation of AI, ML, or NLP systems, covering validation of model outputs, behaviors, and business logic.
- **Technical Skills:**  
  - Understanding of quality engineering principles, defect analysis, root cause investigation, and metric-driven reporting.
  - Experience with test automation frameworks, API testing tools, and integration with CI/CD pipelines.
  - Coding or scripting proficiency (e.g., Python, SQL) for test and data validation automation.
- **Collaboration:**  
  - Proven ability to work cross-functionally and communicate technical quality issues with both technical and non-technical stakeholders.

#### Preferred Additional Experience

- **Specialized AI Testing:**  
  Familiarity with LLM prompt testing, RAG pipelines, synthetic/golden dataset design, or AI output auditing.
- **Responsible AI/Model Governance:**  
  Knowledge of model validation best practices, AI observability, fairness, risk assessment, privacy, or compliance in regulated sectors.
- **Industry Knowledge:**  
  Background working in financial services, wealth management, consulting, or similarly regulated industries.

#### Core Competencies

- Analytical, detail-oriented, and systematic approach to testing
- Creative curiosity focused on AI/model behaviors and uncovering operational risk
- Clear, concise documentation and communication skills
- Strong sense of ownership, reliability, and ethical responsibility

---

**Short Summary / Sample Posting:**
> We’re hiring a Senior AI & Data Quality Engineer to spearhead end-to-end testing for LLM and AI-powered applications across the financial sector. Bring your blend of QA experience, automation acumen, and AI/ML validation skills to a high-impact team delivering trustworthy, business-critical AI solutions.