AI Data Engineer

Role: AI Data Engineer

Location: Rockville, MD (3 days onsite / 2 days remote)
Duration: 6 months (high likelihood of extension)

Need min 10+ years of experience

We are looking for an AI Data Engineer to design and implement data pipelines and retrieval systems for a generative AI platform. This role focuses on ingesting, transforming, and indexing domain-specific data to enable accurate and context-aware AI responses.

You will work closely with AI/agent developers and platform engineering teams to continuously enhance retrieval quality and knowledge coverage.

Key Responsibilities

Data Engineering & ETL

• Design and build scalable ETL pipelines for structured and unstructured data ingestion
• Develop robust workflows for document parsing, transformation, and large-scale data loading
• Implement data validation and quality checks to ensure accuracy and completeness
• Leverage AWS services such as S3, Lambda, Step Functions, OpenSearch, and Bedrock

RAG Pipeline Development & Search Optimization

• Architect and optimize Retrieval-Augmented Generation (RAG) pipelines
• Define document chunking strategies and generate vector embeddings
• Improve retrieval quality through tuning ranking, filtering, and hybrid search approaches
• Evaluate performance using benchmarks and retrieval accuracy metrics
• Experiment with embedding models and retrieval techniques to enhance response quality

Quality Engineering & Testing

• Design test strategies for validating data pipelines and ingestion accuracy
• Develop automated regression tests to monitor retrieval performance
• Build evaluation frameworks to measure precision, recall, and relevance
• Promote test-driven development (TDD) practices

Generative AI & Innovation

• Stay updated with advancements in RAG, embeddings, and retrieval systems
• Explore new approaches such as hybrid search, reranking, and contextual retrieval
• Collaborate with AI developers to ensure high-quality, contextually relevant outputs

Security & Compliance

• Follow secure coding practices, especially when handling sensitive and PII data
• Ensure adherence to organizational security policies and compliance requirements
• Participate in threat modeling and secure system design discussions