Job role: AgentCore – Evaluation & Observability / AgentOps Architect
Duration: 3 months contract
Location: Remote
AWS Certification – Mandatory
Active AWS certification is required.
Core Skills: AWS AgentCore, AgentOps, AI/ML, Evaluation, Observability, Generative AI
We are seeking a senior AgentCore / AgentOps Architect to design and establish enterprise-grade evaluation, observability, and operational standards for AI agents.
The ideal candidate will have strong hands-on experience with AWS AgentCore, Amazon Bedrock, AI/ML, LLM evaluation, agent tracing, observability, and production AI systems.
Key Responsibilities:
- Architect and establish AgentOps, Evaluation, and Observability frameworks for AI agents.
- Define standards for monitoring:
- Agent execution paths
- Model and tool calls
- Token usage
- Latency
- Failures and errors
- Infrastructure and model costs
- Establish evaluation frameworks for:
- Task completion
- Correctness
- Faithfulness
- Safety
- Tool-selection accuracy
- Agent performance and reliability
- Design automated evaluation and regression-testing frameworks for AI agents and LLM applications.
- Build a continuous Observe → Evaluate → Improve → Validate → Deploy lifecycle.
- Establish production monitoring, tracing, alerting, and performance dashboards.
- Integrate AgentOps and evaluation capabilities into CI/CD and MLOps pipelines.
- Define quality gates and validation standards before AI agents are promoted to production.
- Partner with AI/ML, cloud, and engineering teams to operationalize agentic AI solutions.
- Drive best practices for AI reliability, safety, governance, and responsible AI.
Required Skills:
- Strong hands-on experience with AWS AgentCore.
- Strong AI/ML and Generative AI background.
- Experience with Amazon Bedrock and LLM-based applications.
- Strong experience in AgentOps / AI Agent Observability.
- Experience designing LLM/AI evaluation frameworks.
- Experience with agent tracing, telemetry, monitoring, and performance analysis.
- Strong understanding of LLM/agent metrics including correctness, faithfulness, safety, latency, cost, and tool-selection accuracy.
- Experience with automated regression testing for AI/LLM applications.
- Strong Python and API development experience.
- Experience with cloud architecture, CI/CD, and MLOps.
