Sr. Data Engineer


eNGINE LLC
Dice Job Match Score™
🔢 Crunching numbers...
Job Details
Skills
- Apache Kafka
- Confluent
- Databricks
- Microsoft Azure
- Data Warehouse
- DevOps
- Python
- PySpark
- Kafka
- Azure Data Factory
Summary
Senior Data Engineer | Databricks, Azure, Python & Kafka/Confluent
Location: 100% Remote (U.S.)
Work Schedule: Must work Eastern Time (EST) hours
About eNGINE
eNGINE builds Technical Teams. We are a Solutions and Placement firm shaped by decades of interaction with Technical professionals. Our inspiration is continuous learning and engagement with the markets we serve, the talent we represent, and the teams we build. Our Consulting Workforce is encouraged to enjoy career fulfillment in the form of challenging projects, schedule flexibility, and paid training/certifications. Successful outcomes start and finish with eNGINE.
Role Overview
eNGINE is seeking a Senior Data Engineer to help build, evolve, and maintain enterprise data infrastructure supporting a growing data and analytics environment.
This is a hands-on engineering position focused on developing and owning enterprise data warehouse pipelines, data models, and ingestion processes within a Databricks and Azure ecosystem. You'll work extensively with Python, PySpark, SQL, Azure Data Factory, and Azure DevOps to build reliable, scalable data solutions.
A major focus of this role will be advancing the organization's medallion architecture, improving data ingestion patterns, and evolving toward event-driven, low-latency data pipelines. You'll have ownership over assigned data domains, ensuring data quality, pipeline reliability, performance, and documentation standards.
We're looking for an experienced engineer with strong Databricks and Azure expertise and hands-on experience with Confluent or Apache Kafka. Experience with either streaming technology is a must-have for this position. Confluent experience is particularly desirable, but strong Kafka experience will also be considered.
Healthcare, pharmaceutical, or specialty pharmacy experience is a significant advantage, although candidates with strong enterprise data engineering backgrounds in other industries will also be considered.
This is an opportunity to make a meaningful impact on an enterprise data platform while contributing to its continued modernization and growth.
What You'll Be Doing
Enterprise Data Engineering & Architecture
Design, build, and maintain enterprise data warehouse pipelines and data models within a Databricks environment.
Develop and manage data transformations across the Bronze, Silver, and Gold layers of a medallion architecture.
Build and maintain data models, including conformance logic, harmonization, and data survivorship rules.
Take ownership of assigned data domains, ensuring data integrity, pipeline reliability, and adherence to ingestion SLAs.
Help simplify and modernize existing data architectures by consolidating legacy transformation processes into a streamlined medallion architecture.
Data Pipeline Development & Streaming
Develop and maintain production-grade data pipelines using Databricks, Delta Lake, Python, PySpark, and SQL.
Build and optimize data ingestion processes using Azure Data Factory and Azure Data Lake Storage.
Develop and maintain event-driven and low-latency data ingestion architectures.
Leverage Confluent or Apache Kafka to support real-time data ingestion and streaming workflows.
Implement data validation, reconciliation, and quality controls throughout ingestion, transformation, and delivery processes.
Optimize pipeline performance, processing latency, and compute costs across the data platform.
Azure & DevOps Engineering
Utilize Azure DevOps and Git for source control, code reviews, CI/CD, and deployment management.
Develop and maintain automated deployment processes for data pipelines and platform components.
Work with Azure Databricks, Azure Data Factory, and Azure Data Lake Storage to support enterprise data engineering initiatives.
Establish consistent development practices, documentation standards, and operational procedures.
Data Quality, Governance & Observability
Implement data quality checks, validation frameworks, and reconciliation processes to ensure accuracy and consistency.
Monitor pipeline performance, identify anomalies, troubleshoot failures, and drive root cause resolution.
Document data architecture, transformation logic, pipeline dependencies, and operational procedures.
Partner with data governance and QA teams to support data lineage, metadata standards, and pipeline testing.
Implement data contract validation to ensure incoming data meets established requirements before reaching downstream consumers.
Collaboration & Technical Leadership
Partner with Analytics Engineering, Data Governance, QA, and business-facing teams to translate requirements into effective data engineering solutions.
Mentor other Data Engineers on Databricks, PySpark, SQL, and data engineering best practices.
Participate in code reviews and help establish standards for development quality, documentation, and platform reliability.
Work independently to troubleshoot complex data engineering challenges and drive assigned initiatives through completion.
Required Qualifications
6+ years of progressive experience in Data Engineering, including ownership of enterprise data warehouse environments or large-scale data transformation pipelines.
Strong hands-on experience with Databricks in production environments.
Experience with Delta Lake and medallion architecture (Bronze, Silver, Gold).
Advanced SQL skills and strong hands-on proficiency with Python and PySpark.
Experience with Microsoft Azure data services, including:
Azure Databricks
Azure Data Factory
Azure Data Lake Storage
Experience with Azure DevOps, Git, and CI/CD practices.
Hands-on experience with Confluent or Apache Kafka, including event-driven data ingestion, streaming architectures, or real-time data processing.
Strong understanding of enterprise data warehousing, data modeling, transformation logic, and data integration patterns.
Experience building, maintaining, and optimizing production data pipelines.
Understanding of data quality, validation, reconciliation, and pipeline reliability.
Ability to take ownership of complex data engineering initiatives and work effectively with cross-functional teams.
Must be able to work Eastern Time business hours.
Preferred Experience
Healthcare, pharmaceutical, specialty pharmacy, or other regulated industry experience.
Experience with Spark Structured Streaming and event-driven data architectures.
Familiarity with Databricks Unity Catalog, lineage, metric views, and data governance.
Experience with Databricks Zerobus Ingestion or Spark Declarative Pipelines.
Familiarity with data contract standards such as ODCS.
Experience with pipeline observability tools such as Monte Carlo or similar platforms.
Familiarity with enterprise data catalog platforms such as Atlan, Collibra, or equivalent.
Experience mentoring engineers and establishing technical standards across a data engineering team.
Bachelor's degree in Computer Science, Data Engineering, Information Systems, or a related field, or equivalent professional experience.
Next Steps
No C2C, relocation, recruiter represented referral candidates, or sponsorship for this role. For finer details on how eNGINE can impact your career, apply today!
- Dice Id: 91101830
- Position Id: 9148899
- Posted 1 hour ago
Company Info
eNGINE is a Pittsburgh-based Solutions firm specializing in building Technical and Cyber Security teams. We are a trusted business partner to our clients, and provide resources for Project based Consulting, Contract to hire and Direct Hire. We pride ourselves in building lasting relationships with both our consultants as well as our client partners, and aligning the two when the right fit arises. eNGINE applies over 85 years of collective experience representing Consultants, uncovering passive full time hires, and building Technology teams in Pittsburgh. It’s understood that the wrong resource at the right time is worse than no resource at all. You don’t take shortcuts. Neither do we.
