Overview
Remote
Depends on Experience
Contract - Independent
Contract - W2
Contract - 24 Month(s)
Skills
Databricks
Data Modeling
Large Language Models (LLMs)
Machine Learning (ML)
Artificial Intelligence
Job Details
Databricks SME with Data Modeling & AI Agents Expertise
Position Overview
We are seeking a highly skilled Databricks Subject Matter Expert with deep expertise in data modeling and AI agents to lead our data and AI initiatives. This role combines technical leadership, hands-on development, and strategic guidance to drive our organization's data-driven transformation.
Key Responsibilities
Databricks Platform Leadership
- Design and implement scalable data lakehouse architectures using Databricks
- Optimize Delta Lake performance and manage data lifecycle strategies
- Lead Unity Catalog implementation for enterprise data governance
- Develop and maintain complex ETL/ELT pipelines using Spark SQL and PySpark
- Establish best practices for cluster management and cost optimization
- Implement CI/CD workflows for data and ML pipelines
Data Modeling & Architecture
- Design dimensional models (star/snowflake schemas) for analytics workloads
- Implement medallion architecture (bronze/silver/gold) data layers
- Develop data vault models for enterprise data warehousing
- Create and maintain data quality frameworks and validation processes
- Establish master data management strategies
- Design real-time streaming data architectures
AI Agents Development
- Build intelligent agents using large language models (LLMs)
- Implement RAG (Retrieval Augmented Generation) systems
- Design multi-agent workflows and orchestration frameworks
- Develop function calling and tool integration capabilities
- Create conversational AI interfaces for business users
- Implement vector databases and semantic search solutions
Technical Leadership
- Mentor junior data engineers and data scientists
- Conduct technical reviews and architecture assessments
- Establish coding standards and development best practices
- Lead proof-of-concept initiatives for emerging technologies
- Collaborate with stakeholders to translate business requirements into technical solutions
Required Qualifications
Technical Skills
- Databricks Expertise: 3+ years hands-on experience with Databricks platform
- Data Engineering: Advanced Spark SQL, PySpark, and Delta Lake proficiency
- Cloud Platforms: Experience with AWS/Azure/Google Cloud Platform data services
- Programming: Python, SQL, Scala (preferred)
- ML Frameworks: MLflow, scikit-learn, TensorFlow/PyTorch
- Data Modeling: Dimensional modeling, data vault, modern data architecture patterns
- AI/ML: LLMs, vector databases, RAG implementations, agent frameworks
- if Interested please revert back with below details and updated cv.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.