Search Jobs | Dice.com

Job Details

Skills

Azure Data Factory
Pyspark
Azure Cloud
Azure Data Bricks
Azure SQL

Summary

Title: Azure Databricks Lead - Fulltime Role

Only local candidates to NYC, NY or Charlotte, NC

Only one In-Person Interview

Interview Process: Only one round of In-Person interview at any location Charlotte, NC or New York, NY as suitable for applicant.

Job Description:

Data Engineers are responsible for building reliable and scalable data infrastructure that enables organizations to derive meaningful insights, make data-driven decisions, and unlock the value of their data assets.

Job Description - Grade Specific:

We are looking for an experienced Azure Databricks Engineer with strong expertise in cloud-based data engineering, ETL development, and distributed data processing.
The ideal candidate should have solid hands-on experience with PySpark, Delta Lake, Azure Data Factory (ADF) and building scalable data pipelines on Azure.
The engineer will work closely with business stakeholders, Data Architects, and cross-functional teams to design, develop, and optimize data pipelines for enterprise-grade analytics and reporting.

Key Responsibilities:

Design, develop, and optimize ETL/ELT pipelines using Azure Databricks and PySpark.
Build scalable data ingestion workflows from various structured and unstructured data sources.
Implement transformation logic, data cleansing, enrichment, and data validation frameworks.
Work with Delta Lake to build and maintain a Medallion Architecture (Bronze, Silver and Gold layers).
Develop reusable Databricks notebooks and jobs for production-grade data workflows.
Build and orchestrate data pipelines using Azure Data Factory (ADF).
Integrate Databricks with other Azure services, including ADLS, Azure SQL, Event Hubs, Key Vault and Synapse Analytics.
Optimize compute environments, including clusters, pools and auto-scaling configurations.
Implement DevOps processes using Git, CI/CD and Azure DevOps.
Optimize PySpark jobs for performance, scalability and cost efficiency.
Implement best practices for data governance, security and access control.
Troubleshoot production issues and perform root cause analysis.
Conduct code reviews, ensuring adherence to coding standards and data quality requirements.
Collaborate with Data Architects to define architecture standards and design patterns.
Prepare technical documentation, solution diagrams, and operational runbooks.
Collaborate with business stakeholders to understand requirements and translate them into technical solutions.

Mandatory Skills:

Azure Databricks: Notebooks, Jobs, Workflows, Delta Lake, Spark SQL, performance optimization and debugging.
PySpark: DataFrames, transformations, distributed data processing, and optimization.
Azure Data Factory (ADF): Pipelines, triggers, integrations, runtime management, and monitoring.
Azure Data Lake Storage Gen2 (ADLS Gen2): Storage architecture, folder structures, partitioning and security.
Strong understanding of data partitioning strategies and performance tuning.
Experience with security, access control and data governance best practices.
CI/CD implementation using Git, branching strategies and Azure DevOps Pipelines.
Strong SQL skills with proficiency in writing and optimizing complex queries.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 10508656
Position Id: 9033014
Posted 1 day ago

Fitment

Dice Job Match Score™

✨ Finding the perfect fit...

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Senior Data Engineer

New York, New York

•

Today

The Senior Data Engineer is responsible for designing, building, and optimizing enterprise-scale data pipelines and analytics platforms across the Microsoft data ecosystem. This role leads the development of robust, scalable data solutions using Azure Data Factory, Databricks, Microsoft Fabric, SQL Server, and Python to support advanced analytics, machine learning, and business intelligence workloads. The engineer partners closely with architects, data scientists, and business stakeholders to mo

Full-time

USD 175,000.00 - 200,000.00 per year

Data Engineer

New York, New York

•

Today

Job Code 1003 Department Name Health Fund Admin Reports To Manager, Data Engineering FLSA Status Exempt Union Code N/A Management No About Us: Building Services 32BJ Benefit Funds ("the Funds") is the umbrella organization responsible for administering Health, Pension, Retirement Savings, Training, and Legal Services benefits to over 100,000 SEIU 32BJ members. Our mission is to make significant contributions to the lives of our members by providing high quality benefits and services. Thro

Full-time

USD 100,000.00 - 115,000.00 per year

Databricks Data Lake Engineer

Jersey City, New Jersey

•

Today

Job Title: Databricks Data Lake Engineer Location: New York, NY - Jersey City, NJ Role Overview: The Databricks Data Lake Engineer will design, build, and optimize large scale data pipelines across the full lifecycle of data ingestion, migration, curation, and consumption within a modern lakehouse architecture. This role requires hands on expertise with Databricks, Delta Lake, Spark, and cloud-native data platforms, along with the ability to collaborate effectively with business stakeholders, c

Easy Apply

Contract

Azure Databricks Engineer/Architect

Remote

•

7d ago

We are seeking a Solution Architect (Contractor) to design and drive enterprise-scale data engineering architectures on Azure Databricks and cloud platforms. This role will define end-to-end data solutions, architecture patterns, and engineering standards to support analytics, AI/ML, and operational use cases. The ideal candidate will combine deep technical expertise with strong business alignment, enabling scalable, secure, and high-performing data platforms. Solution architecture is critical t

Easy Apply

Contract

Filter Results

Job post features

Posted date

Work settings

Employment type

Distance

Employer type

Work authorization

Azure DataBricks Lead

Skills

Summary

Dice Job Match Score™

Similar Jobs

Azure DataBricks Lead

Job Details

Skills

Summary

Dice Job Match Score™

Similar Jobs