Apply Now

Lead Databricks Engineer with Python (Need Only local to MN)

Eagan, MN, US • Posted 1 day ago • Updated 1 day ago

Contract Corp To Corp

Contract W2

Contract Independent

6 Months

No Travel Required

Able to Sponsor

On-site

Depends on Experience

Fitment

Dice Job Match Score™

🤯 Applying directly to the forehead...

Job Details

Skills

Python
AI
Healthcare

Summary

Title: Senior / Lead Data Engineer

Location: Eagan, MN (Hybrid – 2 days/week onsite)
Duration: 3–6 Months Contract-to-Hire
Start Date: ASAP

Job Summary

We are seeking a Senior/Lead Data Engineer to modernize and own enterprise data pipelines by migrating legacy SAS-based workflows to Python/PySpark on Databricks. The engineer will partner with SMEs during the transition period and eventually take ownership of a critical healthcare analytics pipeline supporting HEDIS reporting.

Key Responsibilities

Migrate legacy SAS pipelines to Python/PySpark on Databricks.
Design, build, and maintain scalable ETL/ELT pipelines.
Develop distributed data processing solutions using Databricks.
Create and optimize complex SQL queries.
Schedule, automate, and monitor data pipelines.
Work with AWS services including S3, Lambda, Glue, and EC2.
Manage Databricks notebooks, workflows, and clusters.
Collaborate with business stakeholders and SMEs to transition pipeline ownership.
Follow Agile methodologies and Git-based development practices.

Required Skills

8+ years of Data Engineering experience.
Strong Python programming and scripting skills.
Hands-on PySpark development.
Extensive Databricks experience (Notebooks, Workflows, Cluster Management).
Experience processing large datasets in distributed environments.
Strong SQL skills.
AWS experience with:
- S3
- Glue
- Lambda
- EC2
Experience building and automating ETL/ELT pipelines.
Git/version control experience.
Agile/Scrum experience.

Preferred Skills

Experience with SAS-to-Python migration.
AI/Automation experience.
Healthcare or HEDIS domain experience.
Lead/Principal Data Engineering experience.

Must-Have Skills

Python
PySpark
Databricks
SQL
AWS (S3, Glue, Lambda, EC2)
ETL/ELT Pipeline Development
Distributed Data Processing
Git
Agile/Scrum

Nice-to-Have Skills

SAS Modernization
AI/Generative AI
Healthcare/HEDIS
Airflow or other workflow orchestration tools
Leadership/Pipeline Ownership experience

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 10120317
Position Id: 9033236
Posted 1 day ago

Contact the job poster

Chandee Pandey

Recruiter! @ Sovereign Technologies

View Profile

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Senior Data Engineer

Hybrid in Eagan, Minnesota

•

Yesterday

Job Description: What are they looking for: Dev Engineer capabilities (this is the core of the role) App dev experience (scripting, especially in python) Expert in AI or Automation (this would be a huge plus) NOTE: they understand this might be a unicorn, so these are listed in priority with 2 and 3 being bonuses Proficient in data manipulation and pipeline development Experience processing large datasets in a distributed environment Hands-on experience with Databricks notebooks, workflows, and

Easy Apply

Contract

70+

Databricks Administrator

Remote

•

7d ago

Job Title: Databricks AdministratorJob SummaryWe are looking for a Databricks Admin with strong hands-on experience in Databricks, AWS, and modern data platforms. The ideal candidate should have experience designing and supporting scalable data pipelines, administering the Databricks platform, optimizing performance and cloud costs, implementing governance, and troubleshooting production issues. Key ResponsibilitiesDesign, build, and maintain scalable data pipelines using AWS and Databricks.Deve

Easy Apply

Full-time

$110,000 - $120,000

Databricks Administrator

Remote

•

7d ago

Easy Apply

Full-time

$110,000 - $120,000

Data Pipeline Engineer (Databricks + AWS Glue)

Remote

•

Today

Location: United States (Remote / Hybrid)Job Type: Full-TimeExperience: 510 Years About the RoleWe are seeking a highly skilled Data Pipeline Engineer with expertise in Databricks, AWS Glue, PySpark, Python, SQL, MongoDB, and AWS to build scalable, cloud-native data platforms. You will be responsible for designing, developing, and optimizing high-performance ETL pipelines that process large volumes of structured and semi-structured data for analytics, reporting, and enterprise applications. This

Easy Apply

Full-time

Depends on Experience

Search all similar jobs

More jobs at Sovereign Technologies in Eagan, MN