AI Engineer

Hybrid in Malvern, PA, US • Posted 2 days ago • Updated 2 days ago
Contract W2
Contract Independent
12 Months
No Travel Required
Hybrid
Depends on Experience
Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Job Details

Skills

  • A/B Testing
  • Amazon Web Services
  • Artificial Intelligence
  • Debugging
  • Regression Testing
  • TypeScript
  • Workflow
  • Orchestration
  • JavaScript
  • Management
  • Optimization
  • Evaluation
  • FOCUS
  • Frontend Development
  • Node.js
  • Prompt Engineering
  • Python
  • Testing
  • UI

Summary

AI Engineer

Work Location: Malvern, PA
Work Arrangement: Onsite – 3 days per week
Pay Rate: $50/hour W2
Assessment: Candidates will be required to complete an online technical test.

Role Overview

We are seeking a hands-on AI Engineer to build and enhance an internal AI-powered developer platform. The primary focus is developing stateful, human-in-the-loop AI agent workflows using LangGraph and Python, along with designing and optimizing automated evaluation frameworks to measure agent performance.

The ideal candidate should have strong experience building production-grade AI agent workflows, implementing deterministic state management, and conducting automated regression testing.

Important: We are not looking for candidates whose experience is limited to basic RAG implementations or simple chatbot development. Candidates should demonstrate strong engineering fundamentals and hands-on experience with LangGraph, agent orchestration, and LLM evaluation frameworks.

Platform Overview

The internal developer platform consists of two major components:

  • Agent Plane: AI agent workflows built using LangGraph and Python.

  • Collaborative UI: A user interface built using Node.js to support human-agent interaction.

Key Responsibilities

  • Design, develop, and debug stateful AI agent workflows using LangGraph.

  • Implement agent state management, checkpointing, interrupts, and tool-calling capabilities.

  • Extend and maintain automated LLM evaluation frameworks.

  • Conduct A/B testing across different agent workflows and prompt variations.

  • Optimize prompts through measurable improvements and regression testing.

  • Evaluate AI agent performance based on response quality, cost, and latency.

  • Develop and enhance human-agent interactive UI capabilities.

  • Ensure reliability, consistency, and performance of AI-driven workflows.

Required Qualifications

  • Strong hands-on experience with LangGraph, including state graphs, checkpointing, interrupts, and tool-calling agents.

  • Experience with LLM evaluation frameworks, including golden datasets, LLM-as-a-Judge, and A/B testing.

  • Strong prompt engineering skills, including structured outputs and prompt optimization.

  • Strong programming experience with Python.

  • Experience developing and testing production-ready AI agent workflows.

  • Ability to measure and improve LLM performance through automated evaluation and regression testing.

Preferred Qualifications

  • Experience with Claude and AWS Bedrock.

  • Front-end development experience using JavaScript/TypeScript.

  • Experience with Tiptap or Y.js.

  • Familiarity with LLM observability tools such as Arize and OpenTelemetry.

  • Experience with Node.js and AWS.

Candidate Submission Details

  • Rate: $50/hour W2

  • Location: Malvern, PA

  • Work Arrangement: Onsite 3 days per week

  • Interview Process: Online technical assessment required.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: 10451163
  • Position Id: 9144748
  • Posted 2 days ago
Contact the job poster
JC

Johnson Carter

Recruiter @ Innovative IT Solutions Inc

Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.