6+ months
Job Title: AI Design/Development Engineer
We have an Immediate Opportunity for an experienced AI Design/Development Engineer on a long-term contract basis. Selected resource will design, build, and deploy an AI-driven alert triage and investigation system on the Claude Platform, integrated with our Splunk environment, and take it from development through production. The work includes building agent workflows that use scoped, read-only Splunk search tools and enrichment sources to review alerts, gather supporting evidence, and return structured verdicts with confidence levels and cited evidence. The contractor will also build an evaluation framework using our historical alerts and analyst dispositions, run the system in shadow mode alongside our analysts, and iterate on prompts and tools until agreed accuracy thresholds are met. Production readiness includes secure deployment, secrets management, logging and tracing of every agent step, monitoring of cost, latency, and error rates, and safeguards against prompt injection from untrusted alert content. Our internal security team will supply domain expertise, including triage criteria, escalation rules, and labeled examples, so the contractor is not expected to bring SOC experience. All prompts, tools, evaluation datasets, architecture documentation, and runbooks will be delivered to us and owned by us. The engagement concludes with knowledge transfer so our team can maintain the system and manage model upgrades.
II. SKILL AND EXPERIENCE REQUIRED:
Claude Platform and agent engineering
Hands-on experience with the Messages API, tool use, and multi-step agent loops. For investigation and enrichment, the agent needs to decide which queries to run, interpret results, and decide when it has enough evidence.
Experience with the Claude Agent SDK or an equivalent orchestration pattern, and with MCP (Model Context Protocol) for exposing Splunk and enrichment sources as tools.
Prompt and context engineering for structured outputs: verdicts, confidence, evidence citations, and recommended next steps in a schema your downstream systems can consume.
Cost and latency management: prompt caching, model tiering (a smaller, faster model for first-pass triage and a stronger one for deep investigation), batching, and token budgeting per alert. Alert volume makes this a first-order design issue.
Ask them which models and features they’ve used recently. The platform changes quickly, so verify against current Anthropic docs rather than trusting anyone’s year-old experience.
Production engineering
Python (most likely), async and queue-based architectures, and idempotent processing. Alerts arrive in bursts, and retries must not create duplicate cases.
Cloud deployment, CI/CD, infrastructure as code, secrets management, and rate-limit and failure handling for both the Claude API and Splunk.
Observability: tracing of every agent step, logging of prompts, tool calls, and outputs, and dashboards for latency, cost, error rate, and verdict distribution.
Evaluation discipline.
· Building a labeled dataset from historical alerts and analyst dispositions.
Measuring false-negative rate (missed true positives) separately from overall accuracy, since a missed real incident is much costlier than an extra escalation.
Regression testing on every prompt or model change, and a plan for model version upgrades.
Shadow-mode deployment, where the AI triages alongside analysts and you compare results before it influences anything.
COMPLETION CRITERIA: The contractor will design, build, and deploy an AI-driven alert triage and investigation system on the Claude Platform, integrated with our Splunk environment, and take it from development through production. The work includes building agent workflows that use scoped, read-only Splunk search tools and enrichment sources to review alerts and return structured verdicts with confidence levels and cited evidence, along with an evaluation framework built from our historical alerts and analyst dispositions. Our internal security team will supply domain expertise, including triage criteria, escalation rules, and labeled examples, so the contractor is not expected to bring SOC experience. Production readiness includes secure deployment, secrets management, full tracing of every agent step, monitoring of cost, latency, and errors, and safeguards against prompt injection from untrusted alert content, with uncertain or failed cases always escalating to a human. The engagement is complete when the system has run in shadow mode for at least 8 weeks and, on a held-out set of historical alerts, meets agreed accuracy thresholds, including a false-negative rate no higher than 5% and agreement with analyst dispositions of at least 90%. Completion also requires passing prompt-injection and read-only access testing, meeting agreed latency and per-alert cost targets, and delivering all source code, prompts, tools, evaluation datasets, documentation, and runbooks, which we will own. Final acceptance is given in writing by our designated internal owner after knowledge transfer, in which our team demonstrates it can run regression tests and deploy a prompt change or model upgrade independently.
