AI Architect - (NPW)

Company Overview

Milestone Technologies is a global IT managed services firm that partners with organizations to scale their technology, infrastructure and services to drive specific business outcomes such as digital transformation, innovation, and operational agility. Milestone is focused on building an employee-first, performance-based culture and for over 25 years, we have a demonstrated history of supporting category-defining enterprise clients that are growing ahead of the market. The company specializes in providing solutions across Application Services and Consulting, Digital Product Engineering, Digital Workplace Services, Private Cloud Services, AI/Automation, and ServiceNow. Milestone culture is built to provide a collaborative, inclusive environment that supports employees and empowers them to reach their full potential.

Our seasoned professionals deliver services based on Milestone's best practices and service delivery framework. By leveraging our vast knowledge base to execute initiatives, we deliver both short-term and long-term value to our clients and apply continuous service improvement to deliver transformational benefits to IT. With Intelligent Automation, Milestone helps businesses further accelerate their IT transformation. The result is a sharper focus on business objectives and a dramatic improvement in employee productivity. Through our key technology partnerships and our people-first approach, Milestone continues to deliver industry-leading innovation to our clients. With more than 3,000 employees serving over 200 companies worldwide, we are following our mission of revolutionizing the way IT is deployed.

Job Overview

Job Overview

We're hiring an AI Architect to design and govern the end-to-end architecture for AI solutions spanning multi-agent orchestration, knowledge-graph reasoning, retrieval-augmented generation (RAG), and evaluation at scale. You will define model strategy, reasoning patterns, data interfaces, and safety/guardrails; establish reusable blueprints; and partner with engineering, security, and data teams to deliver reliable, explainable, and cost-effective AI capabilities.

Key Responsibilities

• AI Architecture & Patterns: Define reference architectures for LLM-powered systems (planner/reasoner, router/composer, tool/connector mesh), including typed I/O contracts and failure isolation.
• Knowledge & Retrieval Strategy: Architect coexistence of Knowledge Graph reasoning and RAG baselines; select embedding/vector stores, chunking, retrieval/reranking, and subgraph/query patterns.
• Model Strategy & Routing: Establish model selection policies (smalllarge escalation), prompt/adapter patterns, caching, and cost/latency budgets; document routing outcomes.
• Evaluation & Quality Gates: Design test sets and scoring rubrics (faithfulness/correctness, precision/recall, multi-hop coverage, latency); implement automated re-evaluation on any change (model/agent/prompt/data).
• Safety & Guardrails: Specify policy-as-code, entitlement checks (role/row/column), PII/PHI handling, and content moderation; define red-team tests and jailbreak defenses.
• Data & Interfaces: Define schemas for AI inputs/outputs; guide ontology/taxonomy alignment; ensure provenance/lineage for KG/RAG pipelines; minimize data movement.
• Operability & Observability: Standardize tracing/logging/metrics for AI runs (call graphs, token/latency/cost); set SLOs and error budgets; partner with DevOps for CI/CD and environment promotion gates.
• Technical Leadership: Review designs/PRs; mentor AI engineers; communicate decisions/trade-offs to stakeholders; maintain decision records and roadmaps.

Required Skills

• Applied LLM Systems: 1+ years in ML/AI or platform architecture with production LLM solutions (planning/reasoning, tool use, function-calling, agent ecosystems).
• Knowledge & Retrieval: Hands-on with Knowledge Graphs (RDF/SPARQL or property graph/Gremlin) and RAG (chunking, embeddings, retrieval/reranking); practical trade-offs between the two.
• Model & Vector Ecosystem: Experience with at least one major model platform (Azure OpenAI, Vertex AI, Anthropic, open-weights) and vector DBs (pgvector, Pinecone, Weaviate, Milvus) plus search (OpenSearch/Elasticsearch).
• Evaluation Engineering: Ability to construct evaluation harnesses, design rubrics, and automate regression testing; familiarity with A/B testing and human-in-the-loop review.
• Security-by-Design: SSO/OIDC, secrets management, least-privilege design, policy-as-code, data minimization, and auditability for AI systems.
• Software & APIs: Strong API design (REST/gRPC), JSON schema contracts, error taxonomies, retries/backoff/idempotency; proficiency in one or more languages (Python, TypeScript/Node.js, Go, or Java).
• Observability & Reliability: OpenTelemetry or equivalent for traces/metrics/logs; resiliency patterns (circuit breakers, bulkheads, backpressure); performance tuning and cost governance.

Good to Have Skills

• Ontology & Graph Practices: SHACL, OWL, ontology stewardship, and data quality checks; graph query optimization and subgraph extraction patterns.
• Prompt & Tooling Ops: Prompt versioning, prompt injection defenses, retrieval parameter tuning, and structured tool-use schemas (e.g., MCP-style adapters).
• MLOps & Platforms: IaC (Terraform), CI/CD for models/prompts/configs, feature flags, canary releases; experience with GPU/accelerator considerations.
• External Tools: Designing safe, auditable action patterns and HITL approvals for external tool execution (simulation/models, document generation, ticketing).
• Cost/Performance Analytics: Token accounting, cache strategies, and per-agent cost ceilings; dashboards for cost-per-answer and latency P50/P95 targets.
• UX for Explainability: Collaborating on rationale/explanation UX so users understand sources, subgraphs, and model/tool decisions.

This role suits a hands-on architect who blends rigorous evaluation and safety with practical engineering patterns-enabling reliable, explainable AI at enterprise scale.

Compensation

Estimated Pay Range:

Exact compensation and offers of employment are dependent on circumstances of each case and will be determined based on job-related knowledge, skills, experience, licenses or certifications, and location.

Our Commitment to Diversity & Inclusion

At Milestone we strive to create a workplace that reflects the communities we serve and work with, where we all feel empowered to bring our full, authentic selves to work. We know creating a diverse and inclusive culture that champions equity and belonging is not only the right thing to do for our employees but is also critical to our continued success.

Milestone Technologies provides equal employment opportunity for all applicants and employees. All qualified applicants will receive consideration for employment and will not be discriminated against on the basis of race, color, religion, gender, gender identity, marital status, age, disability, veteran status, sexual orientation, national origin, or any other category protected by applicable federal and state law, or local ordinance. Milestone also makes reasonable accommodations for disabled applicants and employees.

We welcome the unique background, culture, experiences, knowledge, innovation, self-expression and perspectives you can bring to our global community. Our recruitment team is looking forward to meeting you.