Principal Data Engineer

Hybrid in Pittsburgh, PA, US • Posted 3 hours ago • Updated 3 hours ago
Full Time
75% Travel Required
Hybrid
140000 - 145000/yr
Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Job Details

Skills

  • Databricks
  • Azure Databricks
  • Pyspark
  • Apache Pyspark
  • SQL
  • CI/CD Pipeline
  • Lakehouse Architecture
  • Delta Lake
  • and Medallion Architecture
  • Azure Data Factory.
  • Data Governance
  • Data Processing
  • Microsoft Azure
  • Data Engineering

Summary

About the Opportunity

We are seeking an experienced Data Engineering Manager to lead and support the development of modern, scalable data platforms. This individual will play a key role in designing, building, and optimizing enterprise-level data solutions using Microsoft Azure and Azure Databricks. The ideal candidate will bring strong hands-on data engineering experience, particularly with Azure Databricks, PySpark, Python, SQL, and Lakehouse Architecture. This role is suited for a technical leader who enjoys remaining involved in development while guiding and mentoring data engineering teams. We are also open to experienced Lead Data Engineers, Principal Data Engineers, and Databricks Technical Leads who have strong technical expertise and leadership capabilities.

Key Responsibilities

  • Lead the design, development, and implementation of scalable enterprise data platforms using Azure Databricks.
  • Build, maintain, and optimize batch and streaming data pipelines using Azure Data Factory and other Azure data services.
  • Design and implement Lakehouse Architecture, Delta Lake, and Medallion Architecture.
  • Develop and optimize data processing solutions using PySpark, Python, SQL, and Apache Spark.
  • Improve data pipeline performance, scalability, reliability, and efficiency.
  • Implement data governance, security, and quality standards across enterprise data platforms.
  • Provide technical leadership, mentorship, and guidance to data engineering team members.
  • Collaborate with cross-functional teams to deliver reliable data solutions that support business and analytics needs.

Must-Have Qualifications

  • 5–7+ years of experience in data engineering, data architecture, or related technical roles.
  • 2–4 years of hands-on experience working with Azure Databricks.
  • Strong experience with PySpark, Python, SQL, and Apache Spark.
  • Hands-on experience with Delta Lake, Lakehouse Architecture, and Medallion Architecture.
  • Experience building and managing batch and streaming data pipelines using Azure Data Factory and Azure data services.
  • Strong understanding of data governance, pipeline optimization, and Azure cloud infrastructure.
  • Experience designing and implementing production-level data engineering solutions.

Nice-to-Have Qualifications

  • Experience managing, leading, or mentoring data engineering teams.
  • Knowledge of Scala and PowerShell.
  • Experience with Unity Catalog, Kafka, MLflow, and Apache Airflow.
  • Familiarity with Azure DevOps and CI/CD practices.
  • Relevant certifications, including Azure Data Engineer, Databricks Data Engineer, or Azure Solutions Architect.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: 10371465
  • Position Id: 9145650
  • Posted 3 hours ago
Contact the job poster
Muhammad Jawad

Muhammad Jawad

Recruiter @ CGT Staffing

Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.