Job Description
Job Title: ETL Tester with AI Experience
Location: Chicago, IL (Hybrid 3 days' work from office)
Experience: 10+ Years
Skills: ETL, SQL, Azure Databricks, Agentic AI Testing, Web Testing, Data Automation Tools
We are looking for a senior QA professional skilled in ETL, SQL, Azure Databricks, agentic AI evaluation, web testing, and test automation tools. The ideal candidate will drive data integrity, compliance, and scalable validation strategies.
Roles and Responsibilities Agentic AI Testing and Evaluation
-
Test AI agents that generate extraction queries and transformation logic, along with the workbench through which humans review and approve their outputs.
-
Validate agent-generated metadata executed by the client's existing data platform.
-
Understand and validate the redesigned data processing workflow across raw ingestion, consolidated transformation, and domain-refined layers.
Test Strategy and Reconciliation
-
Own the test strategy and reconciliation methodology.
-
Define equivalence standards, identifying what must match exactly, what may vary within tolerance, and at which layer each applies.
-
Design source-to-target validation across the redesigned data processing workflow.
-
Establish risk-based prioritization, as the schedule does not permit exhaustive coverage of hundreds of tables.
-
Author the test strategy and test plan for review with the client in Phase 2.
Agent Evaluation
-
Build and maintain a golden reference set using existing production extracts, validated with the Epic SME.
-
Maintain strict separation between evaluation data and data used to develop or tune the agents.
-
Establish the evaluation harness in MLflow, including scoring runs, versioning, and candidate comparison.
-
Define and report key metrics, including mapping precision and recall, override rate, confidence calibration, run-to-run consistency, escalation correctness, and cost per case.
-
Operate the regression gate to ensure no prompt, skill, or model change is promoted without passing evaluation.
-
Identify and characterize agent failures, including hallucinated tables, incorrect column selection, omission of required fields, and incorrectly selected join grain.
Automated Regression Testing
-
Own the automated regression harness.
-
Automate comparison of new-path outputs against current production extracts.
-
Validate row counts, aggregate reconciliation, field-level comparisons, data types, precision, null and duplicate profiling, and referential integrity.
-
Establish repeatable execution so every agent revision can be revalidated efficiently.
Data Security and Control Testing
-
Design control-critical test suites for member, health-system, and facility isolation.
-
Validate facility mapping changes and ensure one member cannot access another member's data.
-
Perform contract-scope validation.
-
Verify that generated queries comply with data-sharing restrictions at facility, column, and product levels.
-
Validate acquisition costs by confirming optimized queries access fewer source tables than the baseline and that facility consolidation works as expected.
Team Leadership and Reporting
-
Directly lead two QA engineers.
-
Establish testing standards, review test coverage, and manage defect classification.
-
Confirm downstream compatibility across Databricks, SQL Server, and Vertica.
-
Produce evidence packs suitable for client executive review.
-
Defend testing methodology, sample sizes, and statistical findings when required.
Educational Qualifications
-
Engineering degree BE, ME, BTech, MTech, BSc, or MSc.
-
Technical certifications in multiple technologies are desirable.
Mandatory Skills
-
10+ years of experience in data and ETL testing, including test strategy ownership on at least one large migration or re-platforming program.
-
Proven experience in reconciliation and equivalence testing, demonstrating that rebuilt pipelines produce outputs equivalent to legacy systems.
-
Expert-level SQL skills, including independently writing complex analytical validation queries against unfamiliar schemas.
-
Experience with Azure Databricks and PySpark for large-volume data validation.
-
Python programming for test automation and test harness development.
-
Experience with data quality and test automation frameworks such as Great Expectations, dbt tests, pytest, or equivalent.
-
Experience in test strategy development, coverage models, risk-based prioritization, entry and exit criteria, and defect classification.
-
Multi-platform validation experience across at least two of Databricks, SQL Server, Vertica, or comparable platforms.
-
Experience testing PHI-scoped healthcare data, including de-identified or synthetic test data.
-
Defect management and reporting using Azure Boards.
-
Team leadership experience, including setting testing standards and reviewing team members' test coverage.
-
Strong communication skills and the ability to present and defend quality findings to senior stakeholders.
Preferred Qualifications
-
Experience evaluating LLM or GenAI systems using MLflow, LangSmith, Weights & Biases, or equivalent tools.
-
Experience with LLM-as-judge design and understanding its limitations and failure modes.
-
Applied statistics, including sampling design, confidence intervals, significance testing, and calibration measurement.
-
Epic EHR data experience.
-
Experience with Caboodle or Clarity.
-
Knowledge of clinical coding systems such as ICD, CPT, SNOMED, and LOINC.
-
Experience with multi-tenant data isolation testing.
-
Experience testing code-generation or SQL-generation systems.
-
Previous experience presenting test evidence directly to client executives.
Software Design and Best Practices
-
Apply software design principles such as SOLID and Domain-Driven Design.
-
Understand software code patterns and best practices.
-
Maintain up-to-date technical skills and industry knowledge.
