Senior Data Engineer - Databricks 
Location: [Pittsburgh, PA]
Work Mode: [Onsite]

Job Overview

We are seeking an experienced Senior Data Engineer to design, develop, and maintain secure, scalable, and high-performance enterprise data platforms. The ideal candidate will have strong hands-on experience with Databricks, Apache Spark (PySpark), Python, SQL, Azure Data Factory, Delta Lake, and Apache Kafka.

The candidate will be responsible for building reliable data pipelines, optimizing data processing workflows, implementing data governance and security best practices, and supporting enterprise analytics and reporting initiatives.

Key Responsibilities

• Design, develop, test, deploy, and maintain enterprise-scale data engineering solutions using modern cloud and big data technologies.

• Build and optimize ETL/ELT pipelines using Databricks, Apache Spark (PySpark), Delta Lake, and Azure Data Factory.

• Develop scalable data ingestion, transformation, and integration frameworks for enterprise analytics and reporting.

• Write, optimize, and maintain complex SQL queries, stored procedures, and data transformation logic.

• Design scalable data models to support business intelligence, analytics, and AI initiatives.

• Develop and integrate REST APIs, event-driven architectures, and SOAP services for enterprise data exchange.

• Build and maintain streaming and messaging solutions using Apache Kafka.

• Monitor and troubleshoot production data pipelines, perform root cause analysis, and implement long-term solutions.

• Implement data quality, governance, security, and performance optimization best practices.

• Manage source code using Git and follow CI/CD and DevOps practices.

• Collaborate with architects, product owners, business analysts, and cross-functional engineering teams to deliver technical solutions.

• Participate in Agile ceremonies, architecture discussions, code reviews, and sprint planning.

• Maintain technical documentation, deployment artifacts, testing documentation, and operational runbooks.

• Mentor junior engineers and contribute to engineering standards and continuous improvement initiatives.

Required Skills and Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related technical discipline, or equivalent professional experience.

• 6+ years of experience in data engineering, enterprise data platforms, cloud analytics, or large-scale data processing.

• Strong hands-on experience with:

• Databricks

• Apache Spark and PySpark

• Python

• SQL

• Scala

• Experience with the Databricks ecosystem, including Delta Lake, Unity Catalog, Delta Live Tables (DLT), Databricks SQL, MLflow, Databricks Workspaces, and Databricks Jobs.

• Strong SQL development skills, including query optimization and performance tuning.

• Experience designing and implementing ETL/ELT pipelines using Azure Data Factory or comparable cloud integration tools.

• Experience with Apache Kafka or other event-streaming technologies.

• Experience integrating enterprise applications using REST APIs, JSON, XML, and SOAP web services.

• Experience with Azure Data Lake Storage Gen2 (ADLS Gen2) or comparable cloud storage platforms.

• Proficiency with Git and collaborative source code management.

• Experience working in Linux environments, including shell scripting and command-line utilities.

• Strong analytical, troubleshooting, and problem-solving skills.

• Experience working in Agile environments using Scrum, Kanban, or SAFe methodologies.

• Ability to work independently, manage multiple priorities, and mentor team members.

Preferred Qualifications

• Experience with Microsoft Azure cloud services.

• Familiarity with Terraform or other Infrastructure as Code (IaC) tools.

• Experience implementing CI/CD pipelines using Azure DevOps, GitHub Actions, or similar platforms.

• Exposure to Azure Synapse Analytics, Microsoft Fabric, or Power BI.

• Knowledge of data governance, metadata management, and data quality frameworks.

• Experience with DataOps and MLOps practices.

• Knowledge of enterprise data warehousing and Master Data Management (MDM).

• Experience working in financial services or public sector data environments.

Education

Bachelor's degree in Computer Science, Information Systems, Engineering, or a related discipline, or equivalent professional experience.