Senior AI Engineer
Remote
Client : confidential
Mandatory Skill :Agentic AI; Generative AI; LLMs; LangChain
We are seeking a Senior AI Engineer to design, build, deploy, and operate enterprise-grade Large Language Model (LLM) applications supporting business processes within a regulated financial services environment.
The ideal candidate will have hands-on experience taking Generative AI solutions from design through production deployment and providing ongoing production support. This role requires deep expertise in LLM orchestration, RAG, AI evaluation, Python, cloud services, observability, and production-grade AI engineering.
________________________________________
Key Responsibilities
Design and implement multi-step LLM pipelines, including:
o LLM orchestration
o Structured output generation
o Tool/function calling
o Error handling, retry, and fallback logic
Develop and maintain Retrieval-Augmented Generation (RAG) solutions, including:
o Indexing strategies
o Embedding model selection
o Retrieval quality optimization
Build and maintain LLM evaluation frameworks and golden datasets to measure accuracy, consistency, and regression risk before production releases.
Establish and enforce quality gates for model, prompt, and configuration changes.
Implement observability for non-deterministic AI applications, including:
o Tracing
o Structured logging
o Quality monitoring
o Cost monitoring
Manage cost and latency performance of production AI workloads against defined budgets.
Partner with Product, Business, Risk, Compliance, and Security teams to translate business requirements into technical solutions and support review/approval processes.
Participate in code reviews, architectural decisions, and technical documentation.
Provide ongoing production support for deployed AI applications, including incident response, root-cause analysis, regression management, and remediation.
Mentor junior engineers and contribute to engineering standards and best practices.
________________________________________
Required Qualifications
Bachelor's degree in Computer Science, Engineering, or a related technical field, or equivalent practical experience.
Minimum 5 years of professional software engineering experience.
Minimum 2 years of experience developing LLM/Generative AI applications.
Demonstrated experience deploying at least one enterprise-grade AI application into production and supporting it after launch.
Enterprise Production AI Experience Must Include
AI solution serving real users or business processes, rather than only a proof of concept or internal pilot.
Accountability for non-functional requirements such as:
o Availability
o Latency
o Cost management
o Access control
o Data handling
o Auditability
Experience completing security and compliance reviews within a regulated or enterprise environment.
Post-production ownership involving:
o Incident response
o Regression management
o Changes deployed against live traffic
Technical Requirements
Hands-on experience with at least one LLM agent/orchestration framework, such as:
o LangGraph
o LangChain
o LlamaIndex
o Semantic Kernel
o AutoGen
o Amazon Bedrock Agents
Strong experience with LLM evaluation methodologies, including:
o Dataset/golden dataset curation
o Metric selection
o Evaluation of non-deterministic outputs
o Appropriate use of LLM-as-judge techniques
o Established evaluation frameworks or custom evaluation harnesses
Strong proficiency in Python 3.11+, including:
o Type annotations
o Automated testing
o Asynchronous programming
Experience building and operating cloud-based services such as:
o REST APIs
o Data pipelines
o Event-driven workers
Experience working within codebases governed by automated quality gates and formal code reviews.
________________________________________
Preferred Qualifications
Experience with Document AI / Intelligent Document Processing, including:
o OCR
o Structured field extraction
o Scanned documents
o Photographed documents
o Handwritten source material
Experience designing human-in-the-loop workflows, including:
o Confidence scoring
o Selective automation
o Escalation and review processes
Experience implementing business rules or policies as versioned and reviewable artifacts.
Experience with event-driven architecture, particularly Apache Kafka.
Experience with Kubernetes, Helm, and CI/CD pipeline ownership.
Experience with Infrastructure as Code, particularly Terraform.
Experience with:
o Model selection
o Model benchmarking
o Fine-tuning
o Distillation
o Cost and quality optimization
Prior experience within financial services, insurance, healthcare, or another regulated industry.
________________________________________
Technical Environment
AI / GenAI
LLMs
LLM orchestration frameworks
RAG
Agentic AI
LLM evaluation and observability tooling
Programming / Application Development
Python 3.11+
FastAPI
Pydantic
Data / Database
PostgreSQL with vector extensions
Aurora PostgreSQL
Cloud / AI Platform
AgentCore
Infrastructure / DevOps
Kubernetes
Helm
Terraform
CI/CD
Event-Driven Architecture
Apache Kafka
Observability
Datadog
OpenTelemetry
Key Must-Have Skills for Screening
Python 3.11+ | LLM/GenAI | RAG | LangGraph/LangChain/LlamaIndex/Semantic Kernel/AutoGen/Bedrock Agents | LLM Evaluation | Production AI Deployment & Support | Cloud Services/APIs | AI Observability | Enterprise Security/Compliance
Most important screening point: The candidate should have actually deployed an enterprise-grade GenAI/LLM application to production and supported it post-launch. POC-only GenAI experience would not meet the stated requirement.
