Databricks Engineer with AI / ML

Requirements

We are looking for a skilled and passionate Data Engineer to join our team, with strong expertise in Azure Databricks, AI/ML pipelines, and modern data engineering practices. The ideal candidate will be responsible for designing and building scalable data solutions that power analytics, machine learning, and AI initiatives across the organization.

Key Responsibilities:

• Design, develop, and maintain data pipelines using Azure Databricks, PySpark, and Python.
• Build and optimize data workflows that support AI/ML model training and inference.
• Integrate structured and unstructured data from various sources into a centralized data platform.
• Work closely with Data Scientists and ML Engineers to productionize ML models.
• Ensure data quality, governance, and security across all stages of data processing.
• Participate in the development and implementation of MLOps practices.
• Collaborate with cross-functional teams using Agile methodologies.
• Monitor, debug, and optimize performance of data pipelines in cloud environments.

Required Skills:

• 5+ years of experience in data engineering or a similar role.
• Strong hands-on experience with Azure Databricks and PySpark.
• Proficiency in SQL and Python for data processing.
• Good understanding of AI/ML model lifecycle and integration into pipelines.
• Experience in working with Delta Lake, Lakehouse architecture, or equivalent.
• Familiarity with MLOps tools such as MLflow, DVC, or similar.
• Knowledge of data orchestration tools like Airflow, Data Factory, or similar.
• Hands-on experience with cloud platforms (preferably Azure).
• Exposure to CI/CD for data pipelines and ML models.

Preferred Qualifications:

• Experience in real-time/streaming data processing 
• Exposure to DevOps tools and automation for data and ML.
• Knowledge of data security, compliance, and governance best practices.
• Azure certifications (e.g., DP-203, AI-102) are a plus.