Full Time: Lead Data Engineer

Lead Data Engineer (Hands-On)
Location: Cary, NC | On-site / Hybrid
Experience: 12–18 Years
Employment: Full-Time
Salary: $140K – $145K per annum + Benefits

About the Opportunity
We are looking for a hands-on Lead Data Engineer to support a centralized, AI-first enterprise Data Hub for a global insurance and financial services organization.
The platform is built on Azure Databricks, ingests 150+ inbound data feeds, and distributes data to 35+ downstream systems using a Bronze / Silver / Gold medallion architecture.
AI is embedded across ingestion, canonical mapping, data quality, reconciliation and business-user access.
This is a senior technical leadership role requiring hands-on coding. The successful candidate will own the end-to-end technical design of the data and AI layers, build reference implementations, review production code and deliver production-grade Python, Scala and PySpark solutions.
Candidates must have written or reviewed production code within the past year.
Key Responsibilities
•    Design and implement enterprise lakehouse architecture using Bronze, Silver and Gold data layers
•    Design ADLS Gen2 zones, Delta Lake tables, partitioning, schema evolution and retention strategies
•    Build metadata-driven and parameterized ingestion frameworks for batch, database extracts, CDC and streaming data
•    Develop production-grade Python, Scala and PySpark solutions
•    Work with Azure Event Hubs, Kafka and Spark Structured Streaming
•    Establish coding, testing and PR review standards
•    Troubleshoot production incidents and optimize Spark workloads and cluster performance
•    Implement CI/CD for Databricks and ADF using Azure DevOps, Terraform and Databricks Asset Bundles
•    Build AI-augmented ingestion and source-to-canonical mapping solutions
•    Implement AI-driven data quality, anomaly detection and automated reconciliation
•    Develop synthetic, privacy-preserving test data solutions
•    Contribute to semantic-layer, knowledge-graph and GPT-powered conversational data access capabilities
•    Implement text-to-SQL and semantic retrieval with row- and column-level security
•    Establish governance using Unity Catalog, lineage, access controls and PII standards
•    Participate in architecture and AI governance forums
•    Mentor engineering teams and provide technical leadership
Must-Have Skills & Experience
•    12–18 years of experience in data engineering / data platform delivery
•    Expert-level Python, Scala and PySpark
•    Strong SQL and data modelling skills
•    Deep expertise in Databricks, Delta Lake and Unity Catalog
•    Experience with Databricks Jobs & Workflows, cluster management and performance tuning
•    Strong Azure experience including:
o    ADLS Gen2
o    Azure Data Factory
o    Azure Event Hubs
o    Azure security and governance
•    Proven experience designing and delivering enterprise-scale medallion / lakehouse architectures
•    3+ years of production experience designing and implementing LLM-based systems
•    Strong knowledge of RAG, agentic/tool-calling workflows, embeddings, vector/hybrid retrieval and prompt engineering
•    Hands-on experience with LangChain, LlamaIndex or LangGraph
•    Experience with Azure OpenAI, OpenAI or Databricks Model Serving
•    Experience implementing evaluation frameworks including golden datasets, regression testing, accuracy measurement and hallucination tracking
•    Experience with metadata-driven frameworks, schema inference, profiling, lineage and catalogs
•    Strong experience with CI/CD and IaC using Azure DevOps, Terraform and Databricks Asset Bundles
•    Knowledge of Entra ID, managed identities, RBAC, POSIX ACLs, Key Vault, private endpoints and PII handling
•    Strong technical communication and ability to present architecture to both technical and business stakeholders
Strongly Preferred