Lead Data Engineer (Hands-On)
Location: Cary, NC | On-site / Hybrid
Experience: 12–18 Years
Employment: Full-Time
Salary: $140K – $145K per annum + Benefits
About the Opportunity
We are looking for a hands-on Lead Data Engineer to support a centralized, AI-first enterprise Data Hub for a global insurance and financial services organization.
The platform is built on Azure Databricks, ingests 150+ inbound data feeds, and distributes data to 35+ downstream systems using a Bronze / Silver / Gold medallion architecture.
AI is embedded across ingestion, canonical mapping, data quality, reconciliation and business-user access.
This is a senior technical leadership role requiring hands-on coding. The successful candidate will own the end-to-end technical design of the data and AI layers, build reference implementations, review production code and deliver production-grade Python, Scala and PySpark solutions.
Candidates must have written or reviewed production code within the past year.
Key Responsibilities
• Design and implement enterprise lakehouse architecture using Bronze, Silver and Gold data layers
• Design ADLS Gen2 zones, Delta Lake tables, partitioning, schema evolution and retention strategies
• Build metadata-driven and parameterized ingestion frameworks for batch, database extracts, CDC and streaming data
• Develop production-grade Python, Scala and PySpark solutions
• Work with Azure Event Hubs, Kafka and Spark Structured Streaming
• Establish coding, testing and PR review standards
• Troubleshoot production incidents and optimize Spark workloads and cluster performance
• Implement CI/CD for Databricks and ADF using Azure DevOps, Terraform and Databricks Asset Bundles
• Build AI-augmented ingestion and source-to-canonical mapping solutions
• Implement AI-driven data quality, anomaly detection and automated reconciliation
• Develop synthetic, privacy-preserving test data solutions
• Contribute to semantic-layer, knowledge-graph and GPT-powered conversational data access capabilities
• Implement text-to-SQL and semantic retrieval with row- and column-level security
• Establish governance using Unity Catalog, lineage, access controls and PII standards
• Participate in architecture and AI governance forums
• Mentor engineering teams and provide technical leadership
Must-Have Skills & Experience
• 12–18 years of experience in data engineering / data platform delivery
• Expert-level Python, Scala and PySpark
• Strong SQL and data modelling skills
• Deep expertise in Databricks, Delta Lake and Unity Catalog
• Experience with Databricks Jobs & Workflows, cluster management and performance tuning
• Strong Azure experience including:
o ADLS Gen2
o Azure Data Factory
o Azure Event Hubs
o Azure security and governance
• Proven experience designing and delivering enterprise-scale medallion / lakehouse architectures
• 3+ years of production experience designing and implementing LLM-based systems
• Strong knowledge of RAG, agentic/tool-calling workflows, embeddings, vector/hybrid retrieval and prompt engineering
• Hands-on experience with LangChain, LlamaIndex or LangGraph
• Experience with Azure OpenAI, OpenAI or Databricks Model Serving
• Experience implementing evaluation frameworks including golden datasets, regression testing, accuracy measurement and hallucination tracking
• Experience with metadata-driven frameworks, schema inference, profiling, lineage and catalogs
• Strong experience with CI/CD and IaC using Azure DevOps, Terraform and Databricks Asset Bundles
• Knowledge of Entra ID, managed identities, RBAC, POSIX ACLs, Key Vault, private endpoints and PII handling
• Strong technical communication and ability to present architecture to both technical and business stakeholders
Strongly Preferred
