Tech Lead/Architect - Databricks Data Engineering & AI/ML

EazyML, ( recognized by Gartner, specializes in Responsible AI. Our solutions enable proactive compliance and sustainable automation for enterprises adopting AI at scale. We're also associated with breakthrough startups like Amelia.ai, giving our team exposure to cutting-edge AI products at enterprise scale.

We are seeking an experienced Tech Lead/Architect with strong hands-on expertise in Databricks, Data Engineering, and AI/ML to lead the design, development, and delivery of scalable data and AI solutions. The ideal candidate combines deep technical expertise with proven leadership skills, capable of guiding a team of engineers while remaining actively involved in architecture, coding, and problem-solving.

Key Responsibilities

• Lead the technical design and architecture of large-scale data engineering and AI/ML solutions on Databricks (Lakehouse Platform).
• Provide hands-on technical leadership across the full data lifecycle: ingestion, transformation, storage, and consumption using Spark, Delta Lake, and Databricks workflows.
• Design and implement scalable ETL/ELT pipelines, ensuring performance, reliability, and cost optimization.
• Guide the team in building and deploying AI/ML models and GenAI/LLM-based solutions, integrating them into production data pipelines.
• Collaborate with data scientists, ML engineers, and business stakeholders to translate requirements into robust technical solutions.
• Establish and enforce best practices for data governance, security, CI/CD, and DataOps/MLOps within Databricks environments.
• Mentor and coach a team of data engineers, conducting code reviews and driving engineering excellence.
• Own technical decision-making, including platform architecture, tool selection, and performance tuning.
• Act as the primary technical point of contact for client/stakeholder discussions, ensuring alignment between business goals and technical execution.
• Stay current with evolving Databricks features, Delta Lake updates, and AI/GenAI trends, driving adoption of relevant innovations.

Required Qualifications

• 8+ years of overall experience in data engineering, with 3+ years in a technical leadership or lead role.
• Strong hands-on experience with Databricks, including:
• Spark (PySpark/Scala), Delta Lake, Unity Catalog
• Databricks Workflows/Jobs, Cluster management & optimization
• Databricks SQL and performance tuning
• Solid experience in Data Engineering, including:
• Building and optimizing ETL/ELT pipelines at scale
• Data modeling (batch and streaming)
• Working with cloud data platforms (Azure/AWS/Google Cloud Platform – specify as needed)
• Experience with orchestration tools (e.g., Airflow, ADF, Databricks Workflows)
• Practical experience in AI/ML, including:
• Building, training, and deploying ML models
• Familiarity with GenAI/LLM frameworks (e.g., LangChain, RAG pipelines, vector databases) is a strong plus
• MLOps practices for model lifecycle management
• Strong programming skills in Python and SQL; familiarity with Scala is a plus.
• Experience with CI/CD pipelines and version control (Git-based workflows).
• Solid understanding of data security, governance, and compliance best practices.
• Excellent communication and stakeholder management skills, with the ability to work directly with onshore clients/business teams.
• Experience in the Life Sciences domain (pharma, biotech, clinical, or healthcare data) is preferred.
• Proven experience leading and mentoring technical teams.
• Strong ownership mindset with the ability to balance hands-on delivery and team leadership.
• Excellent problem-solving and analytical thinking.
• Strong communication skills to bridge technical and business conversations.
• Ability to work independently in a fast-paced, onshore client-facing environment.

Preferred Qualifications

• Databricks certification(s) (e.g., Databricks Certified Data Engineer Professional, Databricks Certified Machine Learning Professional).
• Cloud certification (Azure/AWS/Google Cloud Platform).
• Experience in Agile/Scrum delivery environments.
• Prior experience working in client-facing consulting or professional services roles.

Why Join Us

• Join a Gartner-recognized Responsible AI company at the forefront of enterprise GenAI adoption, with the opportunity to work on cutting-edge agentic AI systems alongside a team connected to leading AI ventures like Amelia.ai. This is a fully remote role with the flexibility to work from anywhere in USA, must have authorization to work in US.