Lead AI Engineer (Search Modernization)

Introduction:

We are looking for an AI Engineer to modernize and enhance our existing regex/keyword-based ElasticSearch system by integrating state-of-the-art semantic search, dense retrieval, and LLM-powered ranking techniques. This role will drive the transformation of traditional search into an intelligent, context-aware, personalized, and high-precision search experience.

Responsibilities:

• Modernizing the Search Platform
• Analyze limitations in current regex & keyword-only search implementation on ElasticSearch.
• Enhance search relevance using BM25 tuning, synonyms, analyzers, custom tokenizers, boosting strategies, and scoring optimization.
• Introduce semantic/vector-based search using dense embeddings.
• LLM-Driven Search & RAG Integration
• Implement LLM-powered search workflows including query rewriting and expansion, embedding generation, hybrid retrieval, and re-ranking.
• Build RAG (Retrieval Augmented Generation) flows using ElasticSearch vectors, OpenSearch, or AWS-native tools.
• Search Infrastructure Engineering
• Build and optimize search APIs for latency, relevance, and throughput.
• Design scalable pipelines for indexing structured and unstructured text, maintaining embedding stores, and real-time incremental updates.
• Implement caching, failover, and search monitoring dashboards.
• AWS Cloud Delivery
• Deploy and operate solutions on AWS, leveraging OpenSearch Service or EC2-managed ElasticSearch, Lambda, ECS/EKS, API Gateway, SQS/SNS, and SageMaker.
• Implement CI/CD for search models and pipelines.
• Evaluation & Continuous Improvement
• Develop search evaluation metrics and conduct A/B experiments to measure improvements.
• Tune ranking functions and hybrid search scoring.
• Partner with product teams to refine search behaviors with real usage patterns.

Requirements:

Required Skills & Qualifications:

• 5-10 years of experience in AI/ML, NLP, or IR systems, with hands-on search engineering.
• Strong expertise in ElasticSearch/OpenSearch: analyzers, mappings, scoring, BM25, aggregations, vectors.
• Experience with semantic search, embeddings, vector databases, ANN techniques, LLM-based retrieval, and RAG architectures.
• Proficient in Python; familiarity with Java/Scala is a plus.
• Hands-on AWS experience (OpenSearch, SageMaker, Lambda, ECS/EKS, EC2, S3, IAM).
• Experience building and deploying APIs using FastAPI/Flask and containerizing with Docker.
• Familiar with typical IR metrics and search evaluation frameworks.

Preferred Skills:

• Knowledge of cross-encoder and bi-encoder architectures for re-ranking.
• Experience with query understanding, spell correction, autocorrect, and autocomplete features.
• Exposure to LLMOps/MLOps in search use cases.
• Understanding of multi-modal search (text + images) is a plus.
• Experience with knowledge graphs or metadata-aware search.