Overview
On Site
Full Time
Part Time
Accepts corp to corp applications
Contract - W2
Contract - Independent
Skills
Unstructured Data
Analytical Skill
Advanced Analytics
Use Cases
Job Scheduling
Onboarding
Data Management
EDM
IT Infrastructure
IT Security
Regulatory Compliance
IT Management
Mentorship
Databricks
Data Lake
Storage
Data Modeling
Data Warehouse
Microsoft Azure
Cloud Computing
ADF
Python
PySpark
SQL
Data Engineering
Analytics
Optimization
Performance Tuning
Cost Management
Meta-data Management
Continuous Integration
Continuous Delivery
SANS
Collaboration
Communication
Leadership
Job Details
Role Overview:
We are seeking a highly skilled Senior Databricks Engineer to design, architect, and implement metadata-driven, scalable, and cost-efficient Lakehouse solutions on Azure. The ideal candidate will have strong expertise in Databricks, Delta Lake, and Azure Data Services with the ability to lead best practices, frameworks, and automation for enterprise-scale data platforms.
Key Responsibilities:
- Architect & Design Lakehouse Solutions
- Build scalable, secure, and cost-optimized Lakehouse solutions using Azure Databricks and Delta Lake.
- Leverage Azure Data Lake Storage (ADLS) for structured, semi-structured, and unstructured data integration.
- Ensure support for ACID transactions, schema enforcement, time travel, governance, lineage, and extensibility.
- Data Modeling & Analytics
- Design Lakehouse-based dimensional data models that align with modern data warehousing principles.
- Optimize for scalable, high-performance analytical workloads and advanced analytics use cases.
- Best Practices & Frameworks
- Define and establish best practices for job scheduling, cluster configuration, modular notebook design, and code reusability.
- Lead development of metadata-driven onboarding templates, pipeline frameworks, and automation tools for accelerated source integration and processing.
- Collaboration & Compliance
- Partner closely with Enterprise Data Management (EDM), IT Infrastructure, and IT Security teams to ensure architectural, operational, and compliance standards are met.
- Provide technical leadership and mentorship to junior engineers.
Required Skills & Experience:
- Strong hands-on expertise with Azure Databricks, Delta Lake, and Azure Data Lake Storage (ADLS).
- Proven experience in Lakehouse architecture, data modeling, and modern data warehousing principles.
- Knowledge of Azure cloud services (e.g., ADF, Synapse, Key Vault, Event Hub).
- Proficiency in Python, PySpark, and SQL for data engineering and analytics.
- Experience with cluster optimization, performance tuning, and cost management.
- Strong understanding of metadata-driven pipelines, automation frameworks, and CI/CD practices.
- Excellent collaboration, communication, and leadership skills.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.