We are seeking an AI Engineer to modernize and enhance an existing regex- and keyword-based ElasticSearch platform by integrating advanced semantic search, dense retrieval, and LLM-powered ranking techniques.
This role will lead the transformation of traditional search into an intelligent, context-aware, high-precision, and scalable search experience. The ideal candidate has strong hands-on experience with ElasticSearch/OpenSearch internals, information retrieval (IR), embedding-based search, BM25 optimization, re-ranking strategies, LLM-driven retrieval pipelines, and AWS cloud deployment.
Roles & Responsibilities
1. Modernizing the Search Platform
- Analyze limitations of current regex- and keyword-only search implementations
- Improve search relevance through:
- BM25 tuning and scoring optimization
- Synonyms, analyzers, and custom tokenizers
- Boosting strategies and relevance weighting
- Introduce semantic and vector-based search using dense embeddings
2. LLM-Driven Search & RAG Integration
- Implement LLM-powered search workflows, including:
- Query rewriting and expansion
- Embedding generation (e.g., OpenAI, Cohere, Sentence Transformers)
- Hybrid retrieval (BM25 + vector search)
- Re-ranking using cross-encoders or LLM-based evaluators
- Build Retrieval-Augmented Generation (RAG) pipelines using ElasticSearch vectors, OpenSearch, or AWS-native tooling
3. Search Infrastructure Engineering
- Design and optimize search APIs for latency, relevance, and throughput
- Build scalable pipelines for:
- Indexing structured and unstructured data
- Managing and updating embedding stores
- Supporting real-time and incremental updates
- Implement caching strategies, failover mechanisms, and search monitoring dashboards
4. AWS Cloud Delivery
- Deploy and operate search and AI solutions on AWS using:
- OpenSearch Service or EC2-managed ElasticSearch
- Lambda, ECS/EKS, API Gateway, SQS/SNS
- SageMaker for embedding generation and re-ranking models
- Implement CI/CD pipelines for search models and retrieval workflows
5. Evaluation & Continuous Improvement
- Define and track search quality metrics (nDCG, MRR, precision@k, recall)
- Run A/B tests and relevance experiments
- Continuously tune ranking functions and hybrid scoring logic
- Partner with product teams to refine search behavior using real-world usage data
Required Skills & Qualifications
Experience
- 5 10 years of experience in AI/ML, NLP, or Information Retrieval, with hands-on search engineering
- Deep expertise in ElasticSearch/OpenSearch, including analyzers, mappings, scoring, BM25, aggregations, and vector search
- Proven experience implementing semantic search using:
- Embeddings (BERT, SBERT, LLaMA, GPT-based, Cohere, etc.)
- Vector databases or ElasticSearch vector fields
- Approximate Nearest Neighbor (ANN) techniques
- Working knowledge of LLM-based retrieval and RAG architectures
Technical Skills
- Strong proficiency in Python (Java or Scala experience is a plus)
- Hands-on AWS experience with OpenSearch, SageMaker, Lambda, ECS/EKS, EC2, S3, and IAM
- Experience building APIs using FastAPI or Flask
- Containerization and deployment using Docker
- Familiarity with IR metrics and search evaluation frameworks
Preferred Skills
- Experience with bi-encoder and cross-encoder architectures for re-ranking
- Knowledge of query understanding, spell correction, autocomplete, and query suggestions
- Exposure to LLMOps / MLOps for search and retrieval systems
- Understanding of multimodal search (text + images)
- Experience with knowledge graphs or metadata-aware search systems
Never miss an opportunity! Create an alert based on the job you applied for.
