AI Observability Engineer

Job Title: AI Observability Engineer

Location: Charlotte, NC / Philadelphia, PA

Job Description

We are looking for an experienced AI Observability Engineer to design, deploy, and monitor enterprise AI/ML solutions with a strong focus on Generative AI, LLMs, RAG, and AI observability. The ideal candidate will have hands-on experience building scalable AI applications, implementing monitoring frameworks, and optimizing AI model performance in production environments.

Key Responsibilities

• Design and develop enterprise-grade Generative AI applications using OpenAI, AWS Bedrock, and Hugging Face.
• Build and optimize Retrieval-Augmented Generation (RAG) solutions using LangChain, LangGraph, vector embeddings, and Azure AI Search.
• Develop Agentic AI workflows and multi-agent orchestration solutions.
• Optimize LLM inference using LoRA, QLoRA, vLLM, PagedAttention, and continuous batching.
• Build REST APIs and AI microservices using Python and FastAPI.
• Develop and maintain ML pipelines for model training, deployment, monitoring, and lifecycle management.
• Implement AI observability using tools like Arize to monitor model performance, prompt quality, hallucinations, latency, and inference metrics.
• Establish AI governance, evaluation frameworks, and guardrails for responsible AI.
• Develop machine learning models using PyTorch, Scikit-learn, and XGBoost.
• Build analytics dashboards and provide AI-driven insights to stakeholders.
• Deploy containerized applications using Docker, GitHub, and CI/CD pipelines.
• Collaborate with cross-functional teams to deliver scalable AI solutions.

Required Skills

• Strong experience with Generative AI, LLMs, and RAG architectures.
• Hands-on experience with OpenAI, AWS Bedrock, Hugging Face, LangChain, and LangGraph.
• Proficiency in Python and FastAPI.
• Experience with AI observability platforms such as Arize.
• Knowledge of PyTorch, Scikit-learn, and XGBoost.
• Experience with Docker, GitHub, and CI/CD pipelines.
• Strong understanding of AI governance, model evaluation, and responsible AI practices.
• Excellent analytical, problem-solving, and communication skills.