AI/ML Oversight Specialist

This role is for an AI/ML Oversight Specialist with a strong production software engineering background, specializing in automated testing, regression coverage, pipeline validation, and system quality assurance. The role is technically

hands-on in nature, requiring the Worker to provide independent validation of AI-generated outputs and data pipeline integrity within the RISE data migration program. 

The Worker will rely on their project management and production engineering background to independently validate tooling, architecture, and quality processes.  The resource must have the ability to communicate findings clearly to

conversion specialists, program leadership, and ERS stakeholders. 

Functional Responsibilities:  

ERS is seeking an AI/ML Oversight Specialist with a strong engineering background. The Worker will provide independent quality assurance of architecture decisions, validate outputs and delivery processes within the RISE data

migration program, ensuring the AI capability layer produces accurate, consistent, and auditable results that meet ERS quality and governance standards.

The worker will be responsible for:

• Verify AI/ML architecture decisions and guidelines within the RISE data migration program

• Validate data pipeline integrity processes across high-volume ingestion and transformation processes, including record count reconciliation, field-level accuracy verification, and exception rate tracking against defined quality thresholds

• Review AI pipeline code, test scripts, and data transformation logic across the delivery team, catching data contract issues, performance regressions, and output quality risks before they reach ERS reviewers

• Support knowledge transfer activities by documenting validation frameworks, test coverage maps, and quality assurance processes so ERS-embedded staff can sustain oversight practices post-engagement

• Collaborate with the ML/AI Engineer on pipeline performance optimization and output calibration, applying production systems experience to improve throughput, reduce latency, and ensure data integrity at scale

The Worker should have strong production engineering experience in AI delivery environments and communicate quality findings clearly to non-technical program stakeholders. 

WORKER SKILLS AND QUALIFICATIONS

Minimum:

Skills/Experience

2+ Production software engineering including: 

• Python backend services

• async pipeline architecture 

• gRPC/REST API development 

2+ Demonstrated ability to build, ship, and maintain systems at scale with measurable performance outcomes

2+ Regression testing framework design and execution

2+ Experience building automated test infrastructure that provides coverage across complex multi-system workflows, including systems not accessible via standard DOM or API selectors

2+ High-volume data pipeline validation including batched ingestion, bulk-load integrity verification, record count reconciliation, and exception identification across large structured datasets

1+ Clear technical communication of system behavior, and quality findings to cross-functional teams including engineers, product managers, and non-technical stakeholders in a structured delivery environment

Preferred:

2+ PostgreSQL query optimization, bulk-load performance tuning, and data integrity validation at scale; experience identifying and resolving data quality issues across high-volume ingestion and transformation pipelines

1+ Cross-team code review discipline with demonstrated ability to catch API contract issues, performance regressions, and data integrity risks before production; experience reviewing both backend and frontend pull requests across multi-engineer teams

1+ Experience in a production engineering environment requiring end-to-end ownership of quality outcomes across multiple product teams or customer-facing services; comfort operating across ambiguous, fast-moving technical environments with high accountability