Senior Agentic Platform Engineer

Role: Senior Agentic Platform Engineer

Location: Oaks, PA

Key Skills: Python, API, LLM, OAuth, Azure AI, SQL, OpenTelemetry

Experience: 10+ years

Mode of Hire: Full Time

RESPONSIBILITIES

• Building platform services and libraries: Writing Python packages, services, workers and command-line tooling that are versioned, tested, documented and published to internal registries for other teams to depend on.
• Building identity and policy enforcement: Building the central policy engine and its entitlement model as code with tests, and implementing token acquisition on every runtime, JWT validation and entitlement enforcement against it, together with registration and approval flows through enterprise service management APIs.
• Governing the model path: Authoring API management policies for model and tool traffic, including authentication, quotas, metering and callouts to guardrail services, and maintaining a model catalog as code with provider adapters and failover.
• Building memory, retrieval and tools: Implementing checkpoint and store interfaces on document and vector stores, retrieval interfaces over enterprise search, and Model Context Protocol clients and servers behind a tool gateway.
• Integrating workflows: Building workers that run agents as steps in an orchestration engine, with correlation of identity, trace and user context across engine, worker and agent.
• Proving the path end to end: Building reference and test agents against published contracts, onboarding the first consuming teams, and turning their friction into changes in the library and template.
• Instrumenting everything: Wiring tracing and audit events into every library and service so each agent call is attributable.

SKILLS REQUIRED:

• Experience: 9+ years in software engineering, including 6+ years building and operating backend services in production, with personal ownership of their design, reliability and security. Experience limited to support, maintenance or configuration work does not meet this requirement.
• Backend engineering in Python: Typed, asynchronous Python with production web frameworks and HTTP clients, package design and publishing with semantic versioning and backward compatibility, pytest with fixtures and mocks, structured logging, configuration handling and command-line tooling. Python is the primary language, and the role calls for working productively in other languages, such as TypeScript, Java or C#, when the work requires it, using AI coding tools to accelerate the transition while holding the same engineering standard.
• HTTP and API fundamentals: HTTP semantics, REST design, OpenAPI, server-sent events and streaming, error contracts, idempotency, pagination and versioning, including calling and debugging APIs from the command line.
• Distributed systems: Retries with backoff, timeouts, circuit breakers, idempotency, consistency and partitioning, queues and events, caching and latency budgets, with the ability to trace a request across gateway, service and store and isolate where it fails.
• Identity and access, applied: OAuth 2.0 client credentials and on-behalf-of flows, OpenID Connect, JWT structure and validation (issuer, audience, signature, claims), Entra ID app registrations and service principals, managed identities and workload identity federation, credential chains in the Azure identity libraries, RBAC and Key Vault, including diagnosis of authentication and authorization failures to root cause.
• Agent and LLM development: Production experience with LLM APIs, including chat completions, tool calling, structured output, streaming, embeddings and token accounting. Hands-on development with one or more agent frameworks, such as LangGraph, the OpenAI Agents SDK, Microsoft Agent Framework or the Claude Agent SDK, including state management, checkpointing and memory stores. Implementation of Model Context Protocol clients and servers, including transports, tool schemas and authentication.
• AI gateways and policy: Working knowledge of AI gateways, such as Azure API Management, LiteLLM, Portkey or Kong, including model aliases, quotas, metering and guardrails. Working knowledge of policy as code with a policy engine such as Open Policy Agent.
• Data stores: SQL and relational design. Document or key-value stores such as Cosmos DB, including partitioning, item size limits, time to live and throughput provisioning. Caches such as Redis. Vector or search stores such as Azure AI Search or pgvector, including hybrid search fundamentals.
• Evaluation and testing: Golden datasets, deterministic assertions and LLM-as-judge rubrics with tools such as promptfoo, DeepEval or RAGAS, evaluation as a pipeline gate, contract and integration test suites, and load testing fundamentals.
• Observability: OpenTelemetry concepts, including spans, context propagation and resource attributes, instrumentation with LLM observability platforms such as Langfuse or LangSmith, and structured logging that supports investigation.
• Cloud and delivery: Working knowledge of Azure, including private endpoints, DNS and Log Analytics. Docker and Kubernetes application deployment, including Helm and workload identity, and CI/CD pipelines such as Azure DevOps. Working proficiency in Terraform or comparable infrastructure-as-code tooling.
• AI-native delivery: Fluent use of AI coding agents such as Claude Code, Codex, GitHub Copilot or Cursor to accelerate design, coding, testing and documentation, with every output reviewed, tested and understood before it is submitted.
• Communication: Professional command of spoken and written English, with the ability to explain designs, problems and trade-offs clearly to technical and non-technical audiences, give and receive feedback articulately, restate technical direction accurately, and raise risks early and in writing.
• Technical writing: Authoring design documents, technical specifications, runbooks and decision records that are accurate, concise and usable by engineers, reviewers and AI coding tools.
• Enterprise delivery: Working within security review, open-source approval and private network constraints, and delivering through reviewed pull requests with tests and documentation.

PREFERRED SKILLS

• API gateway policy development: Authoring gateway policies, including expressions, reusable fragments and LLM-specific policies, on a platform such as Azure API Management.
• Enterprise integration: Integrating services with IT service management platforms such as ServiceNow and with API catalogs such as Azure API Center. Familiarity with emerging agent identity models such as Microsoft Entra Agent ID.
• Developer experience: Building internal SDKs or project scaffolds, with tools such as Copier or Cookiecutter, that were adopted by multiple engineering teams. Writing developer documentation for both engineers and AI coding tools, including Agent Skills.
• Workflow engines: Building integrations with workflow orchestration engines such as Flowable, Camunda, Kestra or Temporal, including human tasks and durable execution.
• Data platforms: In-depth experience with managed document and search services such as Cosmos DB or Azure AI Search, or with the search capabilities of cloud data platforms such as Snowflake.
• Regulated industry: Financial services or another regulated industry, where governance is built into tooling rather than applied afterwards.