Apply Now

Senior Data Engineer (Azure & Databricks)

Bloomington, MN, US • Posted 2 days ago • Updated 10 hours ago

Full Time

On-site

USD $60.00 - 85.00 per hour

Fitment

Dice Job Match Score™

🔗 Matching skills to job...

Job Details

Skills

Workflow
Analytics
ADF
Data Flow
Environment Management
Documentation
Architectural Design
Continuous Improvement
Data Engineering
Databricks
PySpark
Unity
SQL
SQL Azure
Microsoft SQL Server
SFTP
DevOps
Continuous Integration
Continuous Delivery
Version Control
Data Analysis
Communication
Conflict Resolution
Problem Solving
Collaboration
Microsoft Azure
Data Governance
RBAC
Data Warehouse
Dimensional Modeling
Business Intelligence
Microsoft Power BI
Apache Spark
Performance Tuning
Management
Artificial Intelligence
Machine Learning (ML)
Use Cases
Financial Services
Recruiting

Summary

**This is a 6+ month contract with our client based out of Bloomington, MN. This is a hybrid role, working in office Tuesday/Thursday. Candidates must be able to work in the US without sponsorship.**

We're looking for a Senior Data Engineer with strong Azure experience, especially in Azure Databricks, Delta Lake, and SQL, to build and scale a medallion-based data platform. This role focuses on designing high-performance, governed data pipelines using PySpark, SQL, and Databricks tools to integrate data from Azure systems, SQL Server Managed Instance, and third-party sources, while partnering closely with analytics teams and business stakeholders. Experience or strong interest in supporting AI/ML use cases is highly valued, with financial-services experience considered a plus but not required.
Responsibilities

Design, develop, and optimize data pipelines in Azure Databricks using PySpark and SQL, applying Delta Lake and Unity Catalog best practices.
Build modular, reusable libraries and utilities within Databricks to accelerate development and standardize workflows.
Implement Medallion architecture (Bronze, Silver, Gold layers) for scalable, governed data zones.
Integrate external data sources via REST APIs, SFTP file delivery, and SQL Server Managed Instance, implementing validation, logging, and schema enforcement.
Utilize parameter-driven jobs and manage compute using Spark clusters and Databricks serverless.
Collaborate with data analytics teams and business stakeholders to understand requirements and deliver analytics-ready datasets.
Monitor and troubleshoot Azure Data Factory (ADF) pipelines (jobs, triggers, activities, data flows) to identify and resolve job failures and data issues.
Automate deployments and manage code using Azure DevOps for CI/CD, version control, and environment management.
Contribute to documentation, architectural design, and continuous improvement of data engineering best practices.
Support the design and readiness of the data platform for AI and machine learning initiatives.

Requirements

Strong expertise with Azure Databricks, including PySpark, Delta Lake, Unity Catalog, and the ability to build reusable libraries, utility notebooks, and parameterized jobs.
Advanced SQL skills with experience working in Azure SQL Database and/or SQL Server Managed Instance.
Experience designing, troubleshooting, and supporting data pipelines using Azure Data Factory.
Proven ability to integrate external data sources, including REST APIs and SFTP.
Working knowledge of Azure DevOps for CI/CD, version control, and parameterized deployments.
Demonstrated experience partnering closely with data analytics teams and business stakeholders, supported by strong communication, problem-solving, and collaboration skills.

Interest or experience in preparing data platforms to support AI and machine learning initiatives.

Nice to Haves

Experience implementing Medallion architecture within governed Azure data environments, including data governance and RBAC.
Familiarity with data warehousing concepts, dimensional modeling, and preparing datasets for BI tools such as Power BI.
Understanding of Spark performance optimization, cluster or serverless compute management, and advanced Delta Lake features.
Hands-on experience preparing datasets to support AI/ML use cases.
Prior experience in the financial-services industry.

Our Vetting Process

At Emergent Staffing, we work hard to find Data Engineers who are the right fit for our clients. Here are the steps of our vetting process for this position:

Application (5 minutes)
Online Assessment (40 minutes)
Initial Phone Interview (30-45 minutes)
Virtual Interview with Hiring team
Onsite Interview
Job Offer!

#EmergentStaffing

#IND44

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: RTX1e9ddd
Position Id: b7a06c6422e1483b6f5350b18fc2975f
Posted 2 days ago

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Senior Data Engineer

Eden Prairie, Minnesota

•

Today

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, and data they need to feel their best. Here, you will find a culture guided by diversity and inclusion, talented peers, comprehensive benefits, and career development opportunities. Come make an impact on the communities we serve as you help us advance heal

Full-time

USD 91,700.00 - 163,700.00 per year

Data Engineer

Remote

•

Today

CTEC is a leading technology firm that provides modernization, digital transformation, and application development services to the U.S. Federal Government. Headquartered in McLean, VA, CTEC has over 300 team members working on mission-critical systems and projects for agencies such as the Department of Homeland Security, Internal Revenue Service, and the Office of Personnel Management. The work we do effects millions of U.S. citizens daily as they interact with the systems we build. Our best-in-

Full-time

Lead Data Engineer

Remote

•

8d ago

VENDORS OR C_2_C CANDIDATES WILL NOT BE CONSIDERED Job Description: As a Lead Data Engineer, you will own the design, development, and operationalization of data pipelines, ingestion frameworks, and Lakehouse engineering patterns across the enterprise data platform. You will serve as the primary hands-on technical lead for data engineering delivery, working closely with data, IT, and business teams and implementation partners to build production-grade pipelines that power analytics, BI, and AI/M

Easy Apply

Full-time

$0 - $0

Jr Data Engineer

Remote

•

Today

Jr. Data Engineer Remote Role 6-Month Contract-to-Hire Compensation: $40 per hour ABOUT THE ROLE Join the Digital Platform Execution team at one of our largest Fortune 200 Energy clients as a Jr. Data Engineer. In this remote, 6-month contract-to-hire position, you will support the development and maintenance of scalable data pipelines and analytics solutions. Working closely with senior engineers and analysts, you will help manage data infrastructure, build robust data workflows, and enable bus

Contract

$30.00 - $40.00

Search all similar jobs

More jobs at Emergent Software in Bloomington, MN