Role : AI Platform Engineer (Guardrails, Observability & Evaluation Infrastructure)
Location : Charlotte NC (100% onsite)
AI Platform Engineer to design and build the foundational components that power enterprise-scale GenAI
applications. This includes data guardrails, model safety tooling, observability pipelines, evaluation harnesses, and
standardized logging/monitoring frameworks. This role is critical for enabling safe, reliable, and compliant AI
development across multiple use cases, teams, and business units. Idea is to create the common platform services
that AI team will build upon. Key Responsibilities1. Guardrails, Safety & Governance
Design and implement data guardrail frameworks (pre-processing, redaction, PII/PHI filtering, DLP
integration, prompt defenses).
Build "Model Armor" components such as:
Input validation & sanitization
Prompt-injection defenses
Harmful content detection & policy enforcement
Output filtering, factchecking, grounding checks
Integrate safety tooling (policy engines, classifiers, DLP APIs/safety models).
Collaborate with Security, Compliance, and Data Privacy teams to ensure frameworks meet enterprise
governance requirements.
2. Observability Frameworks
Build and maintain observability pipelines using tools like Arize AI (tracing, quality metrics, dataset
drift/hallucination tracking, embedding monitoring).
Define and enforce platform-wide standards for:
Tracing LLM calls
Token usage and cost monitoring
Latency and reliability metrics
Prompt/model version tracking
Provide reusable SDKs or middleware for engineering teams to adopt observability with minimal friction.
3. Logging, Monitoring & Telemetry
Design standardized LLM-specific logging schemas, including:
Inputs/outputs
Model metadata
Retrieval metadata
Safety flags
User context and attribution
Build monitoring dashboards for performance, cost, anomalies, errors, and safety events.
Implement alerting and SLOs/SLIs for LLM inference systems.
4. Evaluation Infrastructure
Architect and maintain evaluation harnesses for GenAI systems, including:
RAG evaluation (faithfulness, relevance, hallucination risk)
Summarization/QA evaluation
Human-in-the-loop review workflows
Automated eval pipelines integrated into CI/CD
Support frameworks such as RAGAS, G-Eval, rubric scoring, pairwise comparisons, and test case
generation.
Build reusable tooling for teams to write, run, and track model evaluations.
5. Platform Engineering & Reusable Components
Develop shared libraries, APIs, and services for:
Prompt management/versioning
Embedding pipelines and model wrappers
Retrieval adapters
Common data loaders and document preprocessing
Tool/function schemas
Drive consistency across teams with standards, reference architectures, and best practices.
Review system designs across use cases to ensure alignment to platform patterns.
6. Collaboration & Enablement
Partner with AI engineers, product teams, and data scientists to understand cross-cutting needs and convert
them into reusable platform features.
Create documentation, onboarding guides, examples, and developer tooling.
Provide internal training (brown bags, workshops) on guardrails, observability, and evaluation frameworks.
Required Qualifications Technical Skills
5-10+ years software engineering or ML infrastructure experience.
Strong Python engineering fundamentals (FastAPI, async, typing/Pydantic, testing).
Experience with model safety/guardrails approaches (prompt injection defense, PII redaction, toxicity filters, policy enforcement).
Hands-on with Arize AI, LangSmith, or similar LLM observability platforms.
Experience creating evaluation frameworks using RAGAS, G-Eval, or custom rubric systems.
Strong familiarity with vector databases (Pinecone, Weaviate, Milvus), embeddings, and retrieval pipelines.
Solid understanding of LLM architectures, tokenization, embeddings, context limits, and RAG patterns.
Experience in cloud (Google Cloud Platform preferred), Kubernetes/GE, containers, and CI/CD.
Strong understanding of security, governance, DLP, data privacy, RBAC, and enterprise compliance requirements.
Soft Skills
Strong documentation and communication skills.
Ability to influence engineering teams and standardize best practices.
Comfortable working across multiple stakeholders platform, security, ML engineering, product.
Nice to Have
Experience with LangChain/LangGraph or Llamalndex orchestrations.
Experience with Guardrails.ai, Rebuff, Protect AI, or similar LLM security tooling.
Experience with Google Cloud Platform Vertex AI pipelines, Model Monitoring, and Vector Search.
Familiarity with knowledge graphs, grounding models, fact-checking models.
Building SDKs or developer frameworks adopted across multiple teams.
On-prem or hybrid AI deployment experience.
Never miss an opportunity! Create an alert based on the job you applied for.
