only W2 :: 100% remote :: QA/SDET Lead :: Azure Devops and AI/ML Strong

Experience level: 10 + years

Must have skills: SDET skills, Azure devops, python, pytest test automation and Framework, Al ML practices, RAGAS, LLMS

Work Location: Anywhere in US, Supporting during PST hours.

Onsite/Remote: Remote

Brief JD:

Experience in designing LLM/RAG test automation solutions

Experience in testing for bias, drift, and fairness.

Familiarity with performance metrics (precision, recall, F1, ROC-AUC).

Knowledge of MLflow MLOps framework

Knowledge of tools for synthetic data generation and boundary testing

Knowledge of Azure DevOps for CI/CD pipeline development

Experience in RAG/pipeline evaluation frameworks -

Knowledge of Azure DevOps for CI/CD pipeline development

Experience in RAG/pipeline evaluation frameworks -

pytest: Test automation framework

DeepEval or TruLens: LLM test assertions

RAGAS: RAG-specific metrics

0 Eleuther: LLM evaluation harness

Ο Garak or Promptfoo: LLM red-teaming

Evidently: Drift/performance monitoring

Exposure to explainability frameworks (SHAP, LIME, Captum)

Handson experience in API and Database testing.

Practical knowledge of Databricks, Azure Cloud services land distributed data validation.

Proficiency with Azure DevOps pipelines YAML templates, agent pools, CICD workflows.

Experience in implementing AIML practices in testing e.g., test generation, anomaly detection, log analysis to improve test efficiency and coverage.

Familiarity with Cucumber (BDD) and test reporting frameworks (e.g., Allure).

Strong understanding of integration testing across Databricks streaming jobs, applications