Lead Data Engineer
Talentrix AI INC
Dice Job Match Score™
🔢 Crunching numbers...
Job Details
Skills
- ADF
- Apache Kafka
- Apache Spark
- Artificial Intelligence
- DML
- DevOps
- GL
- Leadership
- LangChain
- LlamaIndex
- Machine Learning (ML)
- Machine Learning Operations (ML Ops)
- Meta-data Management
- POSIX
- Microsoft Azure
- Neo4j
- Ontologies
- RBAC
- SPARQL
- SQL
- Scala
- Terraform
Summary
Lead Data Engineer
Introduction:
This senior hands-on leadership role is for a centralized, AI-first enterprise Data Hub for a global insurance and financial services client on Azure Databricks. The Lead Data Engineer will be responsible for owning the end-to-end technical design of the data and AI layers, building reference implementations for the engineering team, and shipping production-grade Python, Scala, and PySpark code weekly.
Responsibilities:
- Data platform architecture and engineering, including lakehouse architecture design
- Developing metadata-driven ingestion frameworks for various data sources
- Building and maintaining PySpark and Scala Spark jobs, coding standards, and testing
- Implementing CI/CD processes for Databricks and ADF in Azure DevOps
- AI-augmented ingestion and canonical mapping
- Driving AI-driven data quality and anomaly detection
- Creating semantic layers and knowledge graphs with security measures
- Providing governance and leadership, including mentoring engineers
Requirements:
Must-Have Skills & Experience:
- Expert-level Python, Scala, and PySpark skills
- Strong SQL and data modeling abilities
- Proficiency in Databricks, Azure data stack, and LLM-based systems
- Evaluation discipline and hands-on experience with LangChain or LlamaIndex
- Metadata-driven frameworks knowledge
- 12-18 years of total experience in data engineering / data platform delivery
- Proven enterprise-scale delivery of a medallion / lakehouse architecture
- Azure security and governance expertise
- Experience with CI/CD and IaC tools
- Clear technical writing and presentation skills
Strongly Preferred:
- Knowledge of knowledge graphs and ontologies
- Experience with ML-based anomaly detection and financial services domain
- LLMOps / MLOps knowledge
- Relevant certifications and exposure to specific tools
- Dice Id: 91173033
- Position Id: 9108763
- Posted 5 days ago
Company Info
About Talentrix AI INC
In a world where technology is advancing faster than ever, many businesses struggle to keep up—not because they lack ambition, but because they lack the right support.
TALENTRIX AI was founded to bridge that gap. We saw how companies were overwhelmed by the complexity of hiring, the challenge of implementing automation, and the need for reliable operational support.
Our approach is different. We don't just provide services—we become your partner. We take the time to understand your unique challenges, your culture, and your goals. Then we design solutions that fit, implement them with care, and stay with you to ensure success.
