Job Title: Senior Data Engineer Agentic AI & Databricks
Location: Mountain View, CA
Number of Positions: 2
Job Type: Contract
Job Overview
We are seeking an experienced Senior Data Engineer with strong expertise in Python, Databricks, semantic data modeling, and Agentic AI to build reliable data pipelines and business-aware data foundations for AI agents.
The ideal candidate will combine hands-on data engineering expertise with practical experience in Retrieval-Augmented Generation (RAG), knowledge graphs, semantic modeling, and AI agent integration. Experience with LangChain, LangGraph, and Langfuse is highly desirable.
This role focuses on leveraging existing LLMs and enterprise AI platforms to connect enterprise data with intelligent, context-aware agent workflows.
Key Responsibilities
-
Design, build, and maintain scalable batch and streaming data pipelines using Python, SQL, PySpark, and Delta Lake.
-
Leverage Databricks Lakeflow for data ingestion, transformation, and workflow orchestration.
-
Develop semantic data models that define business entities, relationships, terminology, and metrics.
-
Build knowledge graphs to connect and organize information across enterprise systems.
-
Develop contextual graphs that capture business events, historical information, and workflow states to provide relevant context to AI agents.
-
Prepare structured and unstructured data for RAG, semantic search, and graph-based retrieval.
-
Integrate enterprise data sources, retrieval pipelines, and tools with LLM applications using LangChain.
-
Build stateful, multi-step AI agent workflows using LangGraph, including tool calling and human-in-the-loop review.
-
Use Langfuse for tracing, prompt management, evaluation, and monitoring of AI agent interactions.
-
Implement data governance, access controls, metadata management, and lineage using Databricks Unity Catalog.
-
Monitor data quality, data freshness, retrieval accuracy, latency, and operational costs.
-
Ensure the reliability, scalability, security, and performance of production data pipelines and AI integrations.
Required Skills & Qualifications
-
Strong production data engineering experience with Python, SQL, Spark/PySpark, and Databricks.
-
Hands-on experience with Delta Lake, ETL/ELT pipelines, incremental data processing, and data modeling.
-
Practical experience in semantic data modeling, knowledge graphs, entity resolution, and relationship mapping.
-
Understanding of contextual graphs and how business history and workflow state support AI agent decisions.
-
Experience with RAG, embeddings, vector search, semantic search, and integrating enterprise data with AI applications.
-
Hands-on experience with LangChain and LangGraph for retrieval, tool integration, and agent workflow development.
-
Familiarity with Langfuse for tracing, evaluation, and AI observability.
-
Experience with graph technologies such as Neo4j, Amazon Neptune, or RDF-based platforms.
-
Knowledge of data quality, data security, governance, monitoring, and production support.
Technical Skills
-
Programming & Data Engineering: Python, SQL, PySpark, Spark, ETL/ELT
-
Databricks Ecosystem: Databricks, Delta Lake, Lakeflow, Unity Catalog, Metric Views
-
Agentic AI: LangChain, LangGraph, Agent Workflows, Tool Calling, Human-in-the-Loop
-
Semantic Data & Graphs: Semantic Modeling, Knowledge Graphs, Contextual Graphs, Entity Resolution
-
RAG & Retrieval: RAG, Embeddings, Vector Search, Semantic Search, Graph-Based Retrieval
-
AI Observability & Evaluation: Langfuse, MLflow, Tracing, Prompt Management, Evaluation
Preferred Qualifications
-
Experience combining graph-based retrieval with vector search.
-
Experience integrating Databricks with graph databases.
-
Experience building reusable data APIs and tools for AI agents.
-
Familiarity with Databricks AI Search / Vector Search, Agent Bricks, or Knowledge Assistant.
-
Experience supporting enterprise AI agent platforms with evaluation, monitoring, and human review workflows.
Ideal Candidate Profile
The ideal candidate is a strong Databricks Data Engineer who can build production-grade data pipelines and integrate enterprise data with AI agent workflows. Candidates should have practical experience with semantic data, RAG, knowledge graphs, and agent frameworks.
Strong Databricks data engineering expertise is essential, along with hands-on experience integrating AI agents with enterprise data.
