AI Product Analyst
Pay Rate: $50-$65/hour
About the Team and the Environment
This role sits within a centralized AI Engineering and Automation Center of Excellence responsible for delivering enterprise-scale AI and automation solutions across a large, distributed North American organization.
The AI work is real, deployed, and actively supporting business operations. This is not a pilot program or innovation lab.
Current capabilities include:
- Reusable AI skills and plug-ins packaged into governed applications used across multiple business functions
- Retrieval-Augmented Generation (RAG), embeddings, and multi-step workflows operating on governed enterprise data
- A governed AI asset marketplace and Model Context Protocol (MCP) catalog covering intake, publishing, versioning, lifecycle management, and maintenance
- Enterprise-grade evaluation frameworks, golden-set testing, guardrails, monitoring, and logging designed to ensure reliable AI outputs at scale
The organization also operates a mature automation ecosystem alongside its AI initiatives, including:
- Approximately 125 unattended production robots
- More than 200 production automations
- More than 19,000 automation jobs executed every 30 days
- Platforms including UiPath and Microsoft Power Platform
For projects requiring significant engineering investment, such as new backend services, infrastructure implementation, or highly complex integrations, a dedicated Enterprise Engineering organization partners with the AI Engineering team to build and deliver solutions. Once completed, those assets are cataloged, governed, and managed through established operational processes.
What This Role Actually Is
This role serves as the Product Owner for the AI Engineering delivery pipeline.
The successful candidate will lead and direct solution development efforts while partnering closely with engineering teams. This is not a hands-on software engineering role, and the individual in this position will not be writing production code.
Key responsibilities include:
- Managing intake, qualification, prioritization, and opportunity assessment for AI initiatives
- Developing business cases and defining value measurement strategies, including establishing baselines, success metrics, attribution methodologies, and measurable outcomes
- Defining requirements and acceptance criteria for AI and LLM-enabled solutions
- Authoring evaluation specifications, curating golden datasets, and establishing acceptance thresholds used to determine production readiness
- Leading user validation activities, adoption tracking, and value realization assessments after deployment
- Monitoring portfolio health across AI assets, including ownership, lifecycle management, versioning, deprecation, and retirement planning
- Managing governance processes for AI skills, plug-ins, and MCP catalog assets, translating security, privacy, compliance, and regulatory requirements into actionable product requirements
- Building strong partnerships with business leaders, operational stakeholders, and executive teams while maintaining disciplined intake and prioritization processes
Important Distinction
Engineering teams own:
- Evaluation harness implementation
- Retrieval pipelines
- Model serving infrastructure
- Monitoring instrumentation
- Technical delivery
This role owns:
- Defining what success looks like
- Defining evaluation criteria
- Defining acceptance thresholds
- Determining production readiness
- Making the final product recommendation for release
Candidates who view evaluation criteria, testing strategy, or acceptance definitions as purely engineering responsibilities are unlikely to be successful in this position.
Must Have
The following are required screening criteria.
Candidates who do not meet all requirements should not be considered a fit.
1. Product Management Experience
Three or more years of experience in:
- Product Management
- Product Ownership
- Product Analysis
Including ownership of at least one AI or LLM-enabled product successfully deployed to production.
Pilot projects, demonstrations, prototypes, and proof-of-concepts do not satisfy this requirement.
2. AI Evaluation Ownership
Demonstrated ownership of acceptance criteria and evaluation processes for AI or LLM-enabled functionality.
Candidates should be able to clearly explain:
- Evaluation specifications
- Golden datasets
- Acceptance thresholds
that they personally defined.
3. RAG and LLM Evaluation Knowledge
Working knowledge of:
- Retrieval-Augmented Generation (RAG)
- Faithfulness
- Groundedness
- Retrieval Precision
- Retrieval Recall
- Answer Relevance
- Regression Testing across prompt and model versions
Enough depth to guide engineering teams, evaluate outputs, and challenge results without requiring direct coding responsibilities.
4. Business Case Development
Experience creating business cases that include:
- Defined baseline
- Success metric
- Attribution methodology
- Measured outcome
Simply reporting hours saved is not sufficient.
5. Executive and Operational Stakeholder Management
Experience partnering with:
- Operational leaders
- Business stakeholders
- Executive leadership
Including:
- Intake management
- Opportunity scoring
- Portfolio reporting
6. Work Authorization
Must be:
- Authorized to work in the United States on a W-2 basis without sponsorship
- Physically located within the United States for the duration of the engagement
Preferred
The following strengthen a candidate profile but are not required.
Do not reject otherwise qualified candidates solely because these items are absent.
- Familiarity with Model Context Protocol (MCP), AI plug-ins, AI skills, AI asset catalogs, or internal capability marketplaces
- Experience with AI evaluation platforms such as:
- RAGAS
- DeepEval
- Braintrust
- LangSmith
- Arize Phoenix
- Weights & Biases Weave
- OpenAI Evals
- Similar evaluation frameworks
- Familiarity with AI governance frameworks, including:
- NIST AI Risk Management Framework
- NIST Generative AI Profile
- ISO 42001
- Colorado AI Act
- Industry-specific AI governance requirements
- Experience operating in regulated environments where auditability, access controls, compliance, and logging are standard requirements
- Insurance, financial services, healthcare, or similarly regulated industry experience
- Familiarity with enterprise platforms such as:
- Applied Epic
- Vertafore
- Microsoft Dynamics 365
- Salesforce
- Orion
- Workday
- Oracle Fusion
- ACORD data standards
- Experience working within an AI or Automation Center of Excellence (CoE) or similar centralized operating model
- Experience supporting demand governance, portfolio management, and ongoing product rationalization efforts
- Exposure to RPA, UiPath, Microsoft Power Platform, or intelligent automation ecosystems
- Experience using Excel, Power BI, or similar tools for value analysis and portfolio reporting
- Jira or Azure DevOps experience for backlog management and delivery planning
- Bachelor's degree or equivalent practical experience
Certifications
No certification is required for this role, and demonstrated experience delivering successful AI-enabled products carries significantly more weight than certifications.
The following may be viewed as minor positives:
- Reforge
- Product School
- Pragmatic Institute
- Scrum.org Product Owner Certifications
- Microsoft AI Fundamentals (AI-900)
Certification Guidance
Do not reference, prioritize, or screen candidates based on:
These certifications are being retired and are not considered target credentials.
There is currently no broadly recognized individual certification for Anthropic Claude expertise, so candidates should not be evaluated based on possession of one.
Never miss an opportunity! Create an alert based on the job you applied for.