Senior AI Engineer - Remote / Telecommute

Job Title: Senior AI Engineer - Remote / Telecommute

Job Location: Richmond, VA

Job Type: Contract

Pay Range: $65.00hr - $70.00hr

Job Overview:

Please prioritize profiles with deep hands-on experience in Agentic AI, AI evaluation frameworks, Azure AI technologies, and end-to-end AI solution design, implementation, and optimization. Our client is looking for candidates with strong implementation depth and solution-design expertise. Seeking a Senior AI Engineer to design, build, deploy, and operate enterprise-grade Large Language Model (LLM) applications supporting business processes within a regulated financial services environment. The candidate will have hands-on experience taking Generative AI solutions from design through production deployment and providing ongoing production support. This role requires deep expertise in LLM orchestration, RAG, AI evaluation, Python, cloud services, observability, and production-grade AI engineering.

Mandatory Requirements:

• Agentic AI / LLM orchestration with hands-on LangChain, LangGraph, LlamaIndex, Semantic Kernel, AutoGen, or Amazon Bedrock Agents.
• Generative AI / LLM application development, with minimum 2 years developing LLM/GenAI applications.
• RAG including embeddings, indexing strategies, vector retrieval, and retrieval-quality optimization.
• Python 3.11+, including type annotations, automated testing, asynchronous programming, APIs/FastAPI.
• LLM Evaluation, including golden datasets, metrics, non-deterministic output evaluation, and LLM-as-judge/evaluation frameworks.
• Enterprise production AI deployment & post-production support including incident response, regression management, latency/cost management, and live production changes.
• AI Observability & Cloud Services including tracing, structured logging, quality/cost monitoring, REST APIs/data pipelines/event-driven services.
• Enterprise Security & Compliance, including security/compliance reviews, access control, data handling, and auditability.

Domain Experience:

Regulated industry strongly preferred, particularly Financial Services. Insurance and Healthcare are also acceptable preferred backgrounds.

Experience Requirements:

Minimum 5 years software engineering + minimum 2 years LLM/GenAI application development.

Key Responsibilities:

• Design and implement multi-step LLM pipelines, including LLM orchestration, structured output generation, tool/function calling, and error handling, retry, and fallback logic.
• Develop and maintain Retrieval-Augmented Generation (RAG) solutions, including indexing strategies, embedding model selection, and retrieval quality optimization.
• Build and maintain LLM evaluation frameworks and golden datasets to measure accuracy, consistency, and regression risk before production releases.
• Establish and enforce quality gates for model, prompt, and configuration changes.
• Implement observability for non-deterministic AI applications, including tracing, structured logging, quality monitoring, and cost monitoring.
• Manage cost and latency performance of production AI workloads against defined budgets.
• Partner with Product, Business, Risk, Compliance, and Security teams to translate business requirements into technical solutions and support review/approval processes.
• Participate in code reviews, architectural decisions, and technical documentation.
• Provide ongoing production support for deployed AI applications, including incident response, root-cause analysis, regression management, and remediation.
• Mentor junior engineers and contribute to engineering standards and best practices.

Required Qualifications:

• Bachelor s degree in Computer Science, Engineering, or a related technical field, or equivalent practical experience.
• Minimum 5+ years of professional software engineering experience.
• Minimum 2+ years of experience developing LLM/Generative AI applications.
• Demonstrated experience deploying at least one enterprise-grade AI application into production and supporting it after launch.

Enterprise Production AI Experience Must Include:

• AI solution serving real users or business processes, rather than only a proof of concept or internal pilot.
• Accountability for non-functional requirements such as availability, latency, cost management, access control, data handling, and auditability.
• Experience completing security and compliance reviews within a regulated or enterprise environment.
• Post-production ownership involving incident response, regression management, and changes deployed against live traffic.

Technical Requirements:

• Hands-on experience with at least one LLM agent/orchestration framework, such as LangGraph, LangChain, LlamaIndex, Semantic Kernel, AutoGen, or Amazon Bedrock Agents.
• Strong experience with LLM evaluation methodologies, including dataset/golden dataset curation, metric selection, evaluation of non-deterministic outputs, appropriate use of LLM-as-judge techniques, and established evaluation frameworks or custom evaluation harnesses.
• Strong proficiency in Python 3.11+, including type annotations, automated testing, and asynchronous programming.
• Experience building and operating cloud-based services such as REST APIs, data pipelines, and event-driven workers.
• Experience working within codebases governed by automated quality gates and formal code reviews.

Preferred Qualifications:

• Experience with Document AI / Intelligent Document Processing, including OCR, structured field extraction, scanned documents, photographed documents, and handwritten source material.
• Experience designing human-in-the-loop workflows, including confidence scoring, selective automation, and escalation and review processes.
• Experience implementing business rules or policies as versioned and reviewable artifacts.
• Experience with event-driven architecture, particularly Apache Kafka.
• Experience with Kubernetes, Helm, and CI/CD pipeline ownership.
• Experience with Infrastructure as Code, particularly Terraform.
• Experience with model selection, model benchmarking, fine-tuning, distillation, and cost and quality optimization.
• Prior experience within financial services, insurance, healthcare, or another regulated industry.

Technical Environment:

• AI / GenAI: LLMs, LLM orchestration frameworks, RAG, Agentic AI, and LLM evaluation and observability tooling.
• Programming / Application Development: Python 3.11+, FastAPI, and Pydantic.
• Data / Database: PostgreSQL with vector extensions, Aurora PostgreSQL.
• Cloud / AI Platform: AgentCore.
• Infrastructure / DevOps: Kubernetes, Helm, Terraform, and CI/CD.
• Event-Driven Architecture: Apache Kafka.
• Observability: Datadog, OpenTelemetry.

Key Must-Have Skills for Screening:

11+ | LLM/GenAI | RAG | LangGraph/LangChain/LlamaIndex/Semantic Kernel/AutoGen/Bedrock Agents | LLM Evaluation | Production AI Deployment & Support | Cloud Services/APIs | AI Observability | Enterprise Security/Compliance

• Most important screening point: The candidate should have actually deployed an enterprise-grade GenAI/LLM application to production and supported it post-launch. POC-only GenAI experience would not meet the stated requirement.

====

• Medical, Dental, and Vision Insurance
• 401(k) Retirement Plan
• Health Savings Account (HSA)
• Disability Insurance (Short-Term and Long-Term)
• Life and AD&D Insurance
• Paid Sick Leave (where required by applicable state or local law)
• Supplemental Insurance Plans
• Identity Theft Protection
• Pet Insurance
• Employee Wellness Programs
• Employee Assistance Program (EAP)
• Career Growth and Professional Development Opportunities