Lead Software Test Automation Architect | Agentic AI
Houston, TX or Coppell, TX ( Local candidates will be given high priority)
Hybrid 2 days on site
12+ Months Contract
Client will not provide any letter)
Job Description:-
We are seeking a Lead Test Automation Architect to define and lead an enterprise-grade test automation strategy across UI, API, data, and platform layers—while evolving our quality engineering capabilities to support Agentic AI / LLM-based systems (probabilistic outputs, confidence-based validation, hallucination detection, and model evaluation). This role will architect scalable automation frameworks, drive engineering excellence, and accelerate delivery through highly productive, maintainable automation programs.
Key Responsibilities:-
Automation Strategy & Architecture
- Define and own the test automation architecture and multi-layer strategy (UI/API/contract/integration/e2e), aligned to product risk and release cadence.
- Establish automation standards, patterns, and reference implementations (framework structure, page-object/screenplay usage, test data patterns, reporting).
- Drive shift-left testing practices (contract testing, component testing, pre-merge validation) and reduce reliance on fragile end-to-end tests.
Agentic AI / LLM Test Model Alignment
- Design and implement testing approaches for non-deterministic AI outputs, including:
- confidence/threshold-based assertions
- semantic similarity checks and rubric-based validation
- goldens/benchmark datasets and evaluation harnesses
- hallucination and groundedness testing strategies
- regression detection across model/prompt/version changes
- Define AI test coverage across accuracy, robustness, bias/safety (as applicable), performance, and reliability.
- Partner with ML/AI, Product, and Engineering to establish acceptance criteria for AI capabilities and measurable quality gates.
High-Productivity Automation Programs
- Build and scale high-throughput automation projects that are reliable, fast, and easy to extend.
- Improve test execution time, reduce flaky tests, and optimize CI/CD pipelines (parallelization, test selection, retries with guardrails).
- Create actionable, engineering-friendly quality dashboards (pass/fail trends, flake rate, coverage, defect escape metrics).
Tooling, CI/CD, and Quality Gates
- Integrate automation into CI/CD with quality gates (PR checks, nightly suites, release readiness criteria).
- Standardize observability for testing: logs, tracing hooks, screenshots/video, artifacts, and test run diagnostics.
- Define practices for test environments, data management, service virtualization, and dependency control.
Technical Leadership & Mentorship
- Serve as the QE technical authority: coach engineers on automation design, code quality, and best practices.
- Lead design reviews for test architecture and automation PRs; enforce coding standards and maintainability.
- Influence roadmap and delivery by translating risk into clear testing priorities.
Required Qualifications:-
Preferred Qualifications (Strongly Desired)
- Experience testing LLM/Agentic AI features or other probabilistic systems, including:
- evaluation harnesses, benchmark datasets, semantic scoring, and drift detection
- prompt/version regression strategies and structured evaluation reports
- Familiarity with modern quality approaches: test pyramid, contract testing, consumer-driven contracts, risk-based testing, shift-left.
- Knowledge of security/performance testing integration (baseline understanding; hands-on is a plus).
- Experience building reusable automation libraries for multiple teams/products.
Regards,
|
Mayank
Xchange Software Inc.
10 Austin Avenue, 2nd Floor | Iselin, NJ – 08830
Direct Line: -
Hangout-
|
Never miss an opportunity! Create an alert based on the job you applied for.
