Senior Data Engineer (Agentic Retrieval & Memory)

Role:                                  Senior Data Engineer (Agentic Retrieval & Memory)

Location:                          Oaks, PA

Duration :                         12+ Months 

Exp. Required: 10+ years

Need candidate who can work onsite from Day 1 (Hybrid basis)

NO  

Experience: 10+ years overall, including 5+ years hands-on data engineering

RESPONSIBILITIES

• Design and build the agent memory interface across the tiers an agent actually needs: working or session state within a run, short-term conversation history, and long-term memory that persists across sessions. This includes what is written, what is summarized or compacted, what expires, and how state is isolated between users and threads.
• Select and implement the right store for each memory tier: a cache or in-memory store for volatile session state, a document or key-value store for conversation history, and a vector store for semantic long-term recall. Match the store to the access pattern rather than forcing one store to serve every tier.
• Design and build the retrieval interface over the client’s existing enterprise search platform, exposed through the shared SDK so agents query it consistently rather than wiring their own integrations.
• Assess the current backing stores against the workload: partitioning strategy, item and document size constraints, time-to-live and retention, read and write patterns under conversational load, latency inside a live agent loop, and cost at volume.

Skills Required

• 5+ years hands-on data engineering in production environments, covering data modelling, storage design, query performance and the operational behaviour of the stores you choose.
• 3+ years working on agentic or LLM data patterns, with real depth in agent memory: session and conversation state, short-term and long-term memory, summarisation and compaction, expiry and retention, and isolation between users and threads. This is the defining requirement: conventional data engineering alone is not sufficient for this position.
• 3+ years with vector and semantic search, using Azure AI Search, PostgreSQL with pgvector, Elasticsearch, or a comparable vector store, including hybrid search, relevance tuning and index design.
• 3+ years designing NoSQL, document or key-value stores for high-write, low-latency workloads: partitioning and sharding strategy, item and document size constraints, time-to-live and retention, and the read and write patterns of conversational or session-based data.
• 2+ years working with caching or in-memory stores such as Redis or equivalent, for volatile and ephemeral state, including expiry strategy and the trade-offs against durable storage.
• Working knowledge of retrieval-augmented generation, including chunking and embedding strategy and how model choice and chunking affect retrieval quality and cost. You will consume an existing vectorised knowledge base more often than you build one.
• 3+ years Python to production standard, building interfaces or libraries consumed by other engineers rather than scripts.
• 2+ years working on a major cloud platform, including managed data services, identity and access to data stores, and private networking to data services.
• Experience evaluating retrieval and memory quality, using groundedness, relevance or comparable measures, rather than relying on subjective assessment.

Preferred skillset

• Azure data platform experience, including Azure AI Search, Cosmos DB and Azure Storage.
• Experience with managed agent memory services or memory frameworks such as those offered by agent platforms, and a view on when to use them rather than building directly on a store.
• Experience with agent frameworks and how retrieval and memory are consumed inside an agent loop.
• Knowledge graph or entity resolution approaches to long-term or structured memory.
• Experience in financial services or another regulated industry, including data residency, retention and the handling of sensitive data.
• Familiarity with OpenTelemetry or platform observability tooling, particularly tracing retrieval and memory calls inside agent runs.
• Exposure to Model Context Protocol (MCP) or comparable patterns for exposing data sources to agents.
• Experience with data catalogues or lineage tooling in an enterprise setting.