Lead Data Engineer

Job Description:

1. Python, PySpark, Scala and SQL – strong production-level data engineering skills.
2. Azure Databricks – Delta Lake, Spark performance tuning, Jobs/Workflows and Unity Catalog.
3. Azure data services – ADLS Gen2 and Azure Data Factory (ADF).
4. Lakehouse / Medallion Architecture – proven experience implementing Bronze, Silver and Gold layers at enterprise scale.
5. Data Pipeline Engineering – metadata-driven ingestion frameworks, data transformation, data quality and large-scale pipeline optimization.
6. Production AI/LLM Experience – hands-on experience designing and deploying LLM-based solutions, including RAG, embeddings, retrieval and prompt engineering.
7. LLM Frameworks – experience with LangChain, LlamaIndex or LangGraph, plus Azure OpenAI, OpenAI or Databricks Model Serving.
8. Azure DevOps / CI-CD – experience automating data pipeline deployments; Terraform and Databricks Asset Bundles are highly relevant.
9. Technical Leadership – ability to design solutions, write production code, review code and guide data engineering teams.
10. Financial services or insurance data, especially financial reconciliation, general ledger or accounting data.
11. AI-driven data quality, anomaly detection and automated reconciliation.
12. Knowledge graphs, ontologies, semantic layers or enterprise Text-to-SQL solutions.
13. Azure security, data governance, PII protection and fine-grained access controls.
14. MLOps/LLMOps, model evaluation and production AI monitoring.