AI Engineer
Work Location: Malvern, PA
Work Arrangement: Onsite – 3 days per week
Pay Rate: $50/hour W2
Assessment: Candidates will be required to complete an online technical test.
Role Overview
We are seeking a hands-on AI Engineer to build and enhance an internal AI-powered developer platform. The primary focus is developing stateful, human-in-the-loop AI agent workflows using LangGraph and Python, along with designing and optimizing automated evaluation frameworks to measure agent performance.
The ideal candidate should have strong experience building production-grade AI agent workflows, implementing deterministic state management, and conducting automated regression testing.
Important: We are not looking for candidates whose experience is limited to basic RAG implementations or simple chatbot development. Candidates should demonstrate strong engineering fundamentals and hands-on experience with LangGraph, agent orchestration, and LLM evaluation frameworks.
Platform Overview
The internal developer platform consists of two major components:
Key Responsibilities
Design, develop, and debug stateful AI agent workflows using LangGraph.
Implement agent state management, checkpointing, interrupts, and tool-calling capabilities.
Extend and maintain automated LLM evaluation frameworks.
Conduct A/B testing across different agent workflows and prompt variations.
Optimize prompts through measurable improvements and regression testing.
Evaluate AI agent performance based on response quality, cost, and latency.
Develop and enhance human-agent interactive UI capabilities.
Ensure reliability, consistency, and performance of AI-driven workflows.
Required Qualifications
Strong hands-on experience with LangGraph, including state graphs, checkpointing, interrupts, and tool-calling agents.
Experience with LLM evaluation frameworks, including golden datasets, LLM-as-a-Judge, and A/B testing.
Strong prompt engineering skills, including structured outputs and prompt optimization.
Strong programming experience with Python.
Experience developing and testing production-ready AI agent workflows.
Ability to measure and improve LLM performance through automated evaluation and regression testing.
Preferred Qualifications
Experience with Claude and AWS Bedrock.
Front-end development experience using JavaScript/TypeScript.
Experience with Tiptap or Y.js.
Familiarity with LLM observability tools such as Arize and OpenTelemetry.
Experience with Node.js and AWS.
Candidate Submission Details