Senior Data Engineer (Agentic Retrieval & Memory)

Role:                                  Senior Data Engineer (Agentic Retrieval & Memory)

Location:                          Oaks, PA

Key Skill:                           Data Engineering, Agentic AI, LLM, Azure AI, NO SQL, Python

Mode of Hire:                  Full time

Skills Required

•         5+ years hands-on experience in data engineering in production environments, covering data modelling, storage design, query performance and the operational behavior of the stores you choose.

•         3+ years experience in working on agentic or LLM data patterns, with real depth in agent memory: session and conversation state, short-term and long-term memory, summarization and compaction, expiry and retention, and isolation between users and threads. This is the defining requirement: conventional data engineering alone is not sufficient for this position.

•         3+ years experience in with vector and semantic search, using Azure AI Search, PostgreSQL with pgvector, Elasticsearch, or a comparable vector store, including hybrid search, relevance tuning and index design.

•         3+ years experience in designing NoSQL, document or key-value stores for high-write, low-latency workloads: partitioning and sharding strategy, item and document size constraints, time-to-live and retention, and the read and write patterns of conversational or session-based data.

•         2+ years experience in working with caching or in-memory stores such as Redis or equivalent, for volatile and ephemeral state, including expiry strategy and the trade-offs against durable storage.

•         Working knowledge of retrieval-augmented generation, including chunking and embedding strategy and how model choice and chunking affect retrieval quality and cost. You will consume an existing vectorised knowledge base more often than you build one.

•         3+ years experience in Python to production standard, building interfaces or libraries consumed by other engineers rather than scripts.

•         2+ years experience in working on a major cloud platform, including managed data services, identity and access to data stores, and private networking to data services.

•         Experience evaluating retrieval and memory quality, using groundedness, relevance or comparable measures, rather than relying on subjective assessment.