Azure Databricks Data Engineer

Overview

Remote
$50 - $60
Contract - W2
Contract - Independent
Able to Provide Sponsorship

Skills

data engineering
Azure Databricks
Apache Spark
PySpark
SQL
Scala
Azure Data Lake Storage
Azure Data Factory
data warehousing
CI/CD
Git
data security
governance

Job Details

Job Title: Azure Databricks Data Engineer

Remote

Job Description:
We are seeking a skilled Azure Databricks Data Engineer to join our team and lead the development of scalable data pipelines and analytics solutions on the Azure cloud platform. The ideal candidate will have hands-on experience with big data technologies, cloud integration, and data engineering best practices using Azure Databricks and related tools.

Key Responsibilities:
Design, develop, and maintain scalable and reliable ETL/ELT pipelines using Azure Databricks and Apache Spark

Integrate data from various sources (structured and unstructured) into Azure Data Lake Storage (ADLS) or Delta Lake

Optimize data pipelines for performance and scalability in cloud environments

Implement data transformation, cleansing, and validation logic

Collaborate with data scientists, analysts, and other engineers to support data needs

Leverage Delta Lake features for transactional storage and version control (time travel, schema evolution, etc.)

Work with Azure services like Data Factory, Synapse, Azure SQL, Blob Storage, etc.

Monitor, troubleshoot, and ensure data pipeline reliability and data quality

Participate in code reviews, documentation, and CI/CD processes using Git and Azure DevOps

Required Skills and Experience:
8+ years of experience in data engineering or big data development

Hands-on experience with Azure Databricks, Apache Spark, and Delta Lake

Proficiency in PySpark, SQL, and optionally Scala

Strong experience with Azure Data Lake Storage (Gen2), Azure Data Factory, and Synapse Analytics

Solid understanding of data warehousing, data lake architecture, and big data processing

Experience with CI/CD, Git, and deployment automation

Knowledge of data security, governance, and cost optimization in cloud environments

Preferred Qualifications:
Experience with MLflow and machine learning workflows in Databricks

Familiarity with Databricks Unity Catalog and data lineage tools

Azure certifications (e.g., DP-203: Data Engineering on Microsoft Azure) are a plus

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

About Innorev Technologies, Inc