Lead Data Engineer / Architect

Hybrid in Cary, NC, US • Posted 5 hours ago • Updated 5 hours ago
Full Time
No Travel Required
Hybrid
Depends on Experience
Company Branding Image
Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Job Details

Skills

  • Data Governance
  • Code Review
  • Collaboration
  • Continuous Delivery
  • Apache Spark
  • Artificial Intelligence
  • ADF
  • Data Quality
  • Databricks
  • Continuous Integration
  • DevOps
  • Evaluation
  • Access Control
  • Meta-data Management
  • LlamaIndex
  • Machine Learning Operations (ML Ops)
  • IT Management
  • LangChain
  • Financial Services
  • General Ledger
  • Microsoft Azure
  • Extract, Transform, Load
  • Insurance
  • Accounting
  • Data Engineering
  • Finance
  • Terraform
  • Unity
  • Workflow
  • SQL
  • Scala
  • Screening
  • Semantics
  • Prompt Engineering
  • PySpark
  • Python
  • Ontologies
  • Optimization
  • Performance Tuning
  • lakehouse
  • medallion architecture

Summary

Role:                  Lead Data Engineer / ArchitectLocation:          Cary, NC
Work Model: Day 1 onsite; hybrid schedule, Tuesday–Thursday onsite
Experience:      10–15 years

2. Mandatory Skills – Must-Have

 

  1. Python, PySpark, Scala and SQL – strong production-level data engineering skills.
  2. Azure Databricks – Delta Lake, Spark performance tuning, Jobs/Workflows and Unity Catalog.
  3. Azure data services – ADLS Gen2 and Azure Data Factory (ADF).
  4. Lakehouse / Medallion Architecture – proven experience implementing Bronze, Silver and Gold layers at enterprise scale.
  5. Data Pipeline Engineering – metadata-driven ingestion frameworks, data transformation, data quality and large-scale pipeline optimization.
  6. Production AI/LLM Experience – hands-on experience designing and deploying LLM-based solutions, including RAG, embeddings, retrieval and prompt engineering.
  7. LLM Frameworks – experience with LangChain, LlamaIndex or LangGraph, plus Azure OpenAI, OpenAI or Databricks Model Serving.
  8. Azure DevOps / CI-CD – experience automating data pipeline deployments; Terraform and Databricks Asset Bundles are highly relevant.
  9. Technical Leadership – ability to design solutions, write production code, review code and guide data engineering teams.
  10. Financial services or insurance data, especially financial reconciliation, general ledger or accounting data.
  11. AI-driven data quality, anomaly detection and automated reconciliation.
  12. Knowledge graphs, ontologies, semantic layers or enterprise Text-to-SQL solutions.
  13. Azure security, data governance, PII protection and fine-grained access controls.
  14. MLOps/LLMOps, model evaluation and production AI monitoring.

 

Important: This is a hands-on Lead Data Engineer position, not a purely managerial or coordination role. Candidates must demonstrate recent production coding experience.

 

4. Recruiter Screening Checklist

Confirm all of the following before submission:

  • 12–18 years of relevant experience.
  • Visa status and eligibility to meet the applicable compensation terms.
  • H-1B transfer candidate, if applicable, can start upon receipt of the transfer receipt notice.
  • Willing and able to work onsite in Cary, NC, from Day 1, Tuesday–Thursday.
  • Strong hands-on Python, PySpark, Scala and SQL.
  • Strong Azure Databricks, Delta Lake, ADLS Gen2 and ADF experience.
  • Proven enterprise lakehouse / medallion architecture experience.
  • Production experience building and deploying LLM/RAG solutions.
  • Recent production coding and technical leadership experience.
  • Compensation expectations confirmed before submission
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: 91174912
  • Position Id: 9146852
  • Posted 5 hours ago

Company Info

About Aivanta Tech Inc

Aivanta Tech Inc is a forward-thinking software consulting company dedicated to helping businesses innovate, scale, and succeed in the digital era. We specialize in delivering AI-driven solutions, custom software development, and end-to-end technology consulting tailored to meet modern business challenges. Our team combines deep technical expertise with a strong understanding of business processes to build scalable, secure, and high-performance applications. From startups to enterprises, we partner with organizations to transform ideas into impactful digital solutions. At Aivanta Tech Inc, we focus on creating value through intelligent automation, data-driven insights, and cutting-edge technologies. Whether it's developing web and mobile applications, implementing AI solutions, or modernizing legacy systems, we ensure seamless execution and measurable results.

Contact the job poster
MK

Muneera Khatoon

Recruiter @ Aivanta Tech Inc

Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.