Staff Machine Learning Platform Engineer, AI Evaluation

Join Apple Services Engineering to build the next generation of AI evaluation systems. We are seeking a staff machine learning platform engineer to lead the architectural design and development of the high availability services and internal tools powering self-service evaluation at scale. You will partner with researchers to operationalize their innovations, transforming complex workflows into intuitive, developer-first platforms. We are looking for builders who thrive in the ambiguity of new initiatives and are passionate about creating scalable infrastructure.

You will join the engineering team responsible for democratizing AI evaluation across the organization. Your focus will be on developing the developer experience-architecting and implementing the APIs, SDKs, and platform services that turn complex evaluation metrics into simple, self-service calls. You will work hand-in-hand with researchers to operationalize sophisticated measurement techniques, ensuring they scale reliably within our high-availability infrastructure. In this role, you will drive the engineering standards for a new organization, upholding the code quality, automation, and testing rigor required to support the rapid evolution of Generative AI and Agentic systems.

8+ years of hands-on software engineering experience, with a track record of owning the technical direction of a platform or infrastructure domain. \nStrong proficiency in the Python ecosystem (e.g., FastAPI, Pydantic, Pandas). You write production-grade code and lead architectural discussions on day one.\nCustomer Obsession & Product Thinking: You have owned the technical roadmap for an internal platform, presented it to senior stakeholders, and shipped against it. You independently translate vague requirements from other teams into concrete engineering specifications and platform roadmaps.\nDemonstrated experience leading technical partnerships with Data Scientists or Researchers: You have taken research code and shipped it as a production service and built the abstractions, testing frameworks, and deployment pipelines that made the next handoff faster than the last..\nStrong expertise in API Design & Platform Infrastructure: You have designed and owned APIs and SDKs that other developers rely on, with a focus on versioning, backward compatibility, and developer experience at scale.\nOperational excellence background: You have architected and owned CI/CD pipelines, containerization (Docker/Kubernetes), and monitoring (Datadog/Prometheus) for production services, and have been accountable for their reliability.\nBachelors in Computer Science or related field, Masters preferred.

Deep familiarity with AI Evaluation Frameworks: You have built, extended, or contributed to modern evaluation tools like DeepEval, Ragas, TruLens, or LangSmith. You understand how to implement and scale model-based evaluation workflows across a large organization.\nEvaluation Service Deployment: Own the deployment, scaling, and operational health of evaluation services in production - including high-throughput evaluation job orchestration (queueing, prioritization, concurrency, auto-scaling), and defining SLAs for evaluation pipeline latency and availability.\nObservability & Reliability: Experience instrumenting production ML evaluation pipelines including tracking evaluation job throughput, queue depth, judge model latency SLAs, scoring drift over time, and failure modes specific to non-deterministic LLM-based evaluation workflows.\nDeep understanding of Generative AI & Agents: You understand the engineering challenges of relying on LLMs and Agents as software components-specifically managing token economics, handling rate limits, and evaluating non-deterministic, multi-step reasoning capabilities. You have built production systems that depend on these components and have solved these problems at scale.\nBuilder Experience: You have thrived in startup-like environments, navigating high ambiguity to deliver complex technical roadmaps from scratch.