AI Operations Specialist Remote
12 Months
A typical day:
Help the team decide which projects will drive the largest business impact, and make the case for the ones worth doing.
Sit with the team that owns a problem, map how the work gets done today, and work out how AI can do it and where people step in for the hard cases.
Build the AI workflow: write the prompts, set up the pipeline, and keep it at the agreed quality level, watching for drift once it's live.
Build the test sets, measure accuracy, and calibrate LLM judges against human reviewers.
Support product teams in agreeing on what "good enough to launch" means for an AI feature, and bring the evidence for the go/no-go call.
Turn unclear policy into criteria a reviewer can apply consistently, working with product, Legal and Policy.
Work with subject matter experts to write labeling guidelines and produce human-verified labels on real data.
Run labeling at scale with vendor agents, and check their work against agreement metrics and your own spot checks.
Make what you build reusable and document what you learn, so the next team gets to production faster.
Keep our methods current as models, tools and vendor agents change, and retire what stops working.
Own the workstreams you're assigned, from setup through quality checks. Model infrastructure and product roadmaps stay with the teams that own them.
Skills
5-7 years in operations, program management, analytics, consulting or applied AI, where you were responsible for results.
You use AI every day and have built something with it that has actually moved the needle, such as a workflow, an automation or an analysis. You know where it breaks, and you keep pushing it further.
You're comfortable with numbers: accuracy, precision and recall, sampling. SQL and notebooks help. If you haven't used them yet, you'll pick them up quickly with AI assistance.
You're curious about how work actually gets done, including the awkward exceptions, and you've taken a messy process and made it measurable.
You've worked on quality in some form, such as review guidelines, QA or audits.
You write clearly and can work with business leads, PMs and engineers in the same week.
You're fine working with limited direction, and you try things yourself instead of waiting for someone to build them. A technical background isn't required.
You don't need to have designed evaluations or calibrated LLM judges before. We'll teach you, and you'll work with a team that has done it many times.
Preferred experience
Hands-on SQL, Python or API experience.
Trust and safety, content moderation, annotation operations or other human-in-the-loop work at scale.
Fine-tuning models or calibrating LLM judges.
Background in a two-sided marketplace, travel, or another operations-heavy business.
Experience setting up intake and prioritization for a new team.
By applying for this position, you acknowledge and agree that Horizontal Talent may contact you regarding your application using automated technology, including phone calls, SMS/text messages, or email, which may be delivered by our virtual AI recruiter, Alex.
Please apply through this online posting or by visiting our Job Board at Applications will be accepted for 4 weeks. For those that join the team, we offer competitive compensation and benefits including medical, dental, vision, and retirement. Check out all we have to offer and how you can become part of the Horizontal Talent Team.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: 10123528
- Position Id: 76764
- Posted 8 hours ago
