Lead Data Engineer

Must-Have Skills & Experience

• 12–18 years of overall experience in Data Engineering / Data Platform delivery.
• Expert-level Python, Scala, and PySpark with production-ready, modular, and well-tested solutions.
• Strong experience troubleshooting Spark workloads and optimizing large-scale batch and streaming pipelines using Delta Lake.
• Strong SQL and Data Modeling skills, including dimensional/normalized modeling, schema design, and data contracts.
• Deep hands-on expertise with Databricks, including Delta Lake, Unity Catalog, Jobs & Workflows, cluster/pool management, performance tuning, and Model Serving.
• Strong Azure Data experience with ADLS Gen2, Azure Data Factory, and Azure Event Hubs.
• 3+ years of production experience designing and delivering LLM-based solutions, including RAG, agentic/tool-calling workflows, chunking, embeddings, vector/hybrid retrieval, and prompt engineering.
• Hands-on experience with LangChain, LlamaIndex, or LangGraph and at least one provider stack such as Azure OpenAI, OpenAI, or Databricks Model Serving.
• Strong AI evaluation practices including golden datasets, regression testing, accuracy/hallucination tracking, and human-in-the-loop feedback.
• Experience building metadata-driven data frameworks, including schema inference, data profiling, lineage, and data catalogs.
• Proven experience delivering enterprise-scale Medallion / Lakehouse architectures.
• Strong Azure security and governance knowledge: Entra ID, Managed Identities, RBAC, POSIX ACLs, Key Vault, Private Endpoints, and PII handling.
• Experience with CI/CD and Infrastructure as Code, including Azure DevOps, Terraform, Databricks Asset Bundles, and automated data pipeline testing.
• Excellent technical communication skills with the ability to document and present architecture decisions to both technical and non-technical stakeholders.

Strongly Preferred

• Knowledge graphs and ontologies: RDF/SPARQL, Neo4j, and graph modeling.
• Enterprise-scale Text-to-SQL or semantic-layer-based natural language query systems.
• ML-based anomaly detection for time-series or transactional financial data.
• Experience in Financial Services or Insurance, including financial close, GL, subledger, reconciliation, or actuarial data.
• LLMOps / MLOps experience including model/prompt versioning, cost governance, and observability.
• Certifications such as Databricks Data Engineer Professional, Azure DP-203/DP-700, or AZ-305.
• Experience with dbt, Great Expectations, Workday, Prism, or Accounting Center.