Data Engineer_Azure Databricks

Remote • Posted 6 days ago • Updated 4 days ago
Full Time
No Travel Required
Remote
$100,000 - $120,000/yr
Fitment

Dice Job Match Score™

🧠 Analyzing your skills...

Job Details

Skills

  • ETL/ ELT solutions
  • (CDC) and or Change Tracking
  • SQL/ T-SQL/ PL-SQL
  • RBAC
  • Azure Databricks
  • Apache Spark
  • DAB
  • and managing CI/CD

Summary

Data Engineer – Azure Databricks

Overview

Design, build, and maintain scalable batch and streaming data pipelines, ETL workflows, and AI solutions within the Microsoft Azure and Fabric ecosystems. This role bridges cloud data engineering, advanced analytics, and machine learning infrastructure.

 

Core Responsibilities:

Data Engineering & Pipelines

  • Build scalable ETL/ELT pipelines using Azure Data Factory (ADF), Databricks, and Fabric Data Factory.
  • Handle full, incremental, and Change Data Capture (CDC) data ingestion.
  • Manage unstructured and structured data at petabyte scale across ADLS Gen2 and OneLake.

 Architecture & Optimization

  • Implement Lakehouse and Medallion Architectures using Delta Lake (ACID, schema evolution, time travel).
  • Conduct deep performance tuning of Apache Spark internals (partitioning, caching, cluster optimization).
  • Optimize complex SQL/T-SQL queries, window functions, and indexing strategies.

Governance & DevOps

  • Enforce security using Unity Catalog, RBAC, row/column-level security, and data masking.
  • Manage CI/CD pipelines via Git, Azure DevOps/GitHub Actions, and Databricks Asset Bundles (DAB).
  • Deploy Infrastructure as Code (IaC) using Terraform, Bicep, or ARM templates.

 AI & Analytics Integration

  • Build infrastructure for AI applications using Azure Machine Learning, Cognitive Services, and RAG.
  • Collaborate with data scientists to support MLOps, Copilot Studio, and automated testing frameworks.
  • Power downstream analytics by developing Power BI dashboards and advanced DAX models.

 

Technical Skills

Azure Ecosystem:

ADF, Synapse Analytics, ADLS Gen2, Azure SQL DB, Fabric, Storage Accounts

Databricks Suite:

PySpark, Delta Lake, Lakeflow Jobs, Delta Live Tables, Auto Loader, Unity Catalog

Languages:

Python, SQL, T-SQL, Spark SQL, .NET

DevOps & IaC:

Git, Azure DevOps, GitHub Actions, Terraform, Bicep, ARM

AI & QA:

MLOps, Azure OpenAI, Copilot, Automated Testing Frameworks, Cloud Monitoring

BI & Analytics:

Power BI, DAX, SAP BusinessObjects

Cross-Cloud (Bonus) :

AWS core services (queuing, storage tiering, data migration)

Required Qualifications

  • Education: Bachelor’s degree in computer science, IT, or a related quantitative field.
  • Experience: 10+ years of professional cloud data engineering experience.
  • Methodologies: Strong proficiency in Agile/Scrum environments using Jira.
  • Soft Skills: Excellent communication skills to bridge technical delivery with business stakeholder requirements.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: 90765065
  • Position Id: 9026062
  • Posted 6 days ago
Contact the job poster
RS

Ramya Srigadha

Recruiter @ Unikon IT
Create job alert
Set job alertNever miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Remote

5d ago

Easy Apply

Contract

0+

Remote or Boston, Massachusetts

Today

Contract

$DOE

Remote or Reston, Virginia

12d ago

Full-time

USD 98,614.00 - 202,850.00 per year

Remote

Today

Full-time

Search all similar jobs