Sr. ETL / ELT Engineer (Azure Data Engineer)

Hybrid in Minneapolis, MN, US • Posted 30+ days ago • Updated 40 minutes ago

Contract Independent

Occasional Travel Required

Remote

$45 - $70/hr

Fitment

Dice Job Match Score™

⏳ Almost there, hang tight...

Job Details

Skills

Apache Spark
Databricks
Microsoft Azure
ELT
Extract
Transform
Load
Meta-data Management
delta lake
Data Engineering
Change Data Capture
SQL
medallion architecture
schema drift handling
Performance Tuning
Optimization
Data Migration
Streaming

Summary

Experience Level

8–10 years of overall data engineering experience, with at least 4–5 years in cloud-based data platforms and 2–3 years in an architecture or lead design role.

Role Overview

The ETL / ELT Architect will lead the design and governance of scalable, secure, and high-performance data pipelines on Microsoft Azure. This role is responsible for defining enterprise-wide Bronze → Silver → Gold (Medallion) architecture standards, Databricks ETL frameworks, and orchestration patterns supporting both batch and streaming workloads.

The architect will act as the technical authority for ETL design decisions, performance optimization, schema evolution, and reliability across the data platform.

Key Responsibilities

1. ETL / ELT Architecture & Standards

Define and govern Medallion Architecture (Bronze, Silver, Gold) standards across the program.

Establish ELT-first design principles using Azure Databricks and Delta Lake.

Design reusable, metadata-driven ETL frameworks supporting multiple ingestion patterns.

Define ingestion strategies for CDC, full loads, and streaming data from Azure Event Hub and databases.

2. Databricks & Delta Lake Architecture

Design and implement Databricks Auto Loader for scalable ingestion with schema drift handling.

Define merge and upsert strategies using Delta Lake for Silver and Gold layers.

Establish best practices for:

Schema evolution and validation

Late-arriving data handling

Idempotent processing

Define Delta Lake maintenance strategies (OPTIMIZE, VACUUM, Z-ORDER).

3. Performance & Optimization

Define partitioning strategies based on data volume, access patterns, and downstream usage.

Optimize Spark workloads for joins, aggregations, and large-scale transformations.

Ensure efficient cluster sizing and job configuration for cost and performance balance.

4. Orchestration & Workflow Design

Define orchestration approaches using Azure Data Factory and Databricks Workflows.

Design dependency management across Bronze, Silver, and Gold pipelines.

Enable parameterized and reusable pipelines supporting multi-tenant and multi-source ingestion.

5. Error Handling, Monitoring & Reliability

Define standardized error handling, retry, and recovery mechanisms.

Implement data quality checks and validation at each layer.

Design observability using Azure Monitor and Alerts.

Ensure pipeline resilience and operational stability.

6. Governance & Downstream Enablement

Align ETL design with Azure security, governance, and lineage standards (Microsoft Purview).

Design Gold-layer data models optimized for Synapse Dedicated SQL Pool, reporting, and analytics.

Support secure data sharing through Azure Data Share and external consumption platforms.

Required Skills & Experience

1. Experience

8–10 years in data engineering and ETL/ELT development.

4+ years designing and implementing cloud-based data platforms (Azure preferred).

2+ years in an architecture, lead, or technical design role.

2. Technical Skills

Strong expertise in Azure Databricks architecture and Spark-based ETL.

Deep hands-on experience with Delta Lake (MERGE, schema evolution, ACID guarantees).

Experience with Databricks Auto Loader for streaming and incremental ingestion.

Proven experience designing enterprise-grade ETL frameworks.

Strong knowledge of schema drift handling, CDC patterns, and incremental processing.

Hands-on experience with Azure Data Factory for orchestration.

Expertise in performance tuning and optimization for Databricks and Spark workloads.

Experience with real-time and streaming data pipelines.

Exposure to data migration and legacy system decommissioning programs.

Strong understanding of error handling, retry logic, and fault-tolerant pipeline design.

3. Cloud & Data Platform

Strong experience with Azure ADLS Gen2, Event Hub, Databricks, Synapse.

Familiarity with Microsoft Purview or equivalent governance tools.

Experience supporting downstream analytics, reporting, and data sharing use cases.

Soft Skills

Strong architectural thinking and decision-making ability.

Ability to define standards and mentor engineering teams.

Excellent communication and documentation skills.

Experience collaborating with platform, security, and analytics stakeholders.

Nice to Have

Knowledge of CI/CD and DevOps practices for Databricks and data pipelines.

Experience working in large enterprise or multi-domain data programs.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: PTPh0RNVYeqzYHw
Position Id: 8860644
Posted 30+ days ago

Sr. ETL / ELT Engineer (Azure Data Engineer)

Dice Job Match Score™

Job Details

Skills

Summary

Similar Jobs