Data Engineer - Databricks

Pittsburgh, PA, US • Posted 4 hours ago • Updated 4 hours ago
Contract W2
24 Weeks
On-site
$120000/yr
Company Branding Image
Fitment

Dice Job Match Score™

📊 Calculating match score...

Job Details

Skills

Summary

Data Engineer – Azure Databricks
Contract-to-Hire
Pittsburgh, PA – Onsite


Job ID J0726-0569

Visa : (No Sponsorship)
Position Overview

  • Seeking an experienced Data Engineer to join a high-performing data engineering team responsible for building modern, cloud-native data platforms on Microsoft Azure.
  • This role will focus on designing and implementing scalable, secure, and high-performance data solutions using Azure Databricks, Apache Spark, PySpark, SQL, Delta Lake, and Unity Catalog. The position requires strong expertise in enterprise data governance, metadata management, security, and access control within the Investment Banking, Securities, or Capital Markets domain.
  • The ideal candidate is passionate about cloud data engineering, enjoys solving complex data challenges, and thrives in a collaborative Agile environment.

Required Competencies
CompetencyImportanceProficiency Level
  • Apache Spark
  • Required
  • Expert
  • Microsoft Azure
  • Required
  • Advanced
  • Investment Banking and Securities
  • Required
  • Advanced
  • Structured Query Language – SQL
  • Required
  • Expert
  • Unity Catalog
  • Required
  • Advanced
  • Databricks
  • Required
  • Expert
Key Responsibilities
  • Design, develop, and maintain scalable data pipelines using Azure Databricks.
  • Build and optimize ETL and ELT workflows using PySpark, SQL, Apache Spark, and Delta Lake.
  • Implement and manage Unity Catalog for enterprise data governance, metadata management, security, and fine-grained access control.
  • Develop cloud-native data engineering solutions on the Microsoft Azure platform.
  • Collaborate with architects, business analysts, and cross-functional teams to translate business requirements into technical solutions.
  • Optimize Spark jobs and data pipelines for performance, scalability, and cost efficiency.
  • Ensure data quality, reliability, governance, and regulatory compliance through validation, monitoring, and engineering best practices.
  • Support production deployments, troubleshooting, root-cause analysis, and performance tuning.
  • Participate in code reviews, technical documentation, and continuous improvement initiatives.
  • Apply cloud data engineering, security, CI/CD, and DevOps best practices within Azure environments.
Required Qualifications
  • 8+ years of overall data engineering or related technology experience preferred.
  • Minimum 5 years of hands-on Azure Databricks experience.
  • Mandatory hands-on experience implementing and managing Unity Catalog.
  • Expert-level experience with Apache Spark, PySpark, and SQL.
  • Strong experience developing enterprise-scale data engineering solutions on Microsoft Azure.
  • Strong experience with Delta Lake, ETL/ELT development, and data pipeline implementation.
  • Experience with Azure Data Factory and Azure Data Lake Storage Gen2.
  • Strong knowledge of the Microsoft Azure data ecosystem.
  • Strong understanding of data governance, metadata management, data security, and access control.
  • Mandatory experience within Investment Banking, Securities, or Capital Markets.
  • Experience using Git and implementing CI/CD practices.
  • Experience working in Agile/Scrum delivery environments.
  • Excellent analytical, communication, and problem-solving skills.
  • Must be willing to work onsite in Pittsburgh, PA.
  • Must be willing and eligible to convert to full-time employment after six months.
Preferred Qualifications
  • Databricks Certified Data Engineer Associate or Professional certification.
  • Microsoft Certified Azure Data Engineer Associate, DP-203.
  • Experience with Infrastructure as Code using Terraform or Bicep.
  • Experience implementing DevOps practices for Azure Databricks environments.
  • Familiarity with Microsoft Purview or Azure Purview.

 


#M1
#DI-CB2


#L1 - KB1

Ref: #404-IT Pittsburgh
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: 10295162
  • Position Id: 377507
  • Posted 4 hours ago

Company Info

About System One

System One is a leading provider of specialized, highly technical outsourced services, workforce solutions, staffing, and recruiting to critical infrastructure, technology, life sciences, and government sectors As an essential partner to private and public organizations of various needs and sizes, we offer our expertise to help them bring their most complex and mission-critical programs to fruition.

With our highly specialized services, in-demand technical skills, and vast operational expertise, we can deliver accelerated results that benefit our clients and the job seekers who partner with us. We staff contract, contract-to-hire, and direct-hire professionals across our network to work with our clients, some of whom are industry leaders and nationally branded organizations.

Our dedicated team of recruiters work tirelessly to match jobseekers with careers that align with their skills, aspirations, and goals.

We understand that finding the right job can be a transformative experience, and we are committed to supporting jobseekers throughout their career journeys. Our recruitment services are designed to connect individuals with fulfilling jobs that offer growth and development opportunities.

Whether you're an organization seeking customized workforce solutions, outsourced services, or if you're a jobseeker searching for your next opportunity,

System One is the perfect staffing partner for you.

About_Company_One
Contact the job poster
Karan Bhatia

Karan Bhatia

Recruitment Manager @ System One
Create job alert
Set job alertNever miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Pittsburgh, Pennsylvania

3d ago

Easy Apply

Contract

$120000

Pittsburgh, Pennsylvania

4d ago

Easy Apply

Contract

Depends on Experience

Pittsburgh, Pennsylvania

16d ago

Easy Apply

Full-time

$100,000 - $130,000

Pittsburgh, Pennsylvania

26d ago

Easy Apply

Contract

$170

Search all similar jobs