We are seeking a hands-on Principal Data Engineer to help re-engineer an enterprise data platform and support the migration of validated workloads from Azure Synapse Analytics to Databricks. This role is ideal for someone who enjoys reverse-engineering complex solutions, strengthening data quality and governance, and ensuring business-critical data flows remain accurate throughout modernization.
Responsibilities
- Assess and re-engineer existing data platform components across ingestion, transformation, storage, and consumption layers.
- Review current pipeline logic, data models, orchestration patterns, and governance controls to identify gaps or technical debt.
- Validate end-to-end data processing behavior, including change data capture, slowly changing dimensions, deduplication, and reconciliation routines.
- Support the migration of workloads to a Databricks lakehouse environment while preserving business logic and control framework behavior.
- Build and refine pipelines using modern lakehouse patterns and ensure output consistency during parallel-run and cutover activities.
- Recreate ingestion, transformation, and orchestration solutions on target platforms with strong auditability and quality checks.
- Collaborate with business, architecture, and governance partners to confirm rules, controls, and data requirements.
- Contribute to code reviews, technical mentoring, and planning for legacy component retirement.
Skills
- 7+ years of hands-on data engineering experience, including work on complex enterprise platforms or migration initiatives.
- Strong experience with Azure Synapse Analytics, including pipelines, Spark, and dedicated SQL pools.
- Solid understanding of ADLS Gen2, Delta Lake, and modern lakehouse architecture patterns.
- Hands-on knowledge of Databricks, PySpark, Spark SQL, and Delta Lake capabilities such as MERGE, CDC, schema enforcement, and time travel.
- Advanced Python and SQL skills for building and supporting ETL/ELT solutions at scale.
- Deep familiarity with data governance, audit, reconciliation, control frameworks, and data quality validation.
- Experience working with enterprise data patterns such as SCD Type 0/1/2, soft and hard deletes, and retroactive changes.
- Ability to analyze existing implementations, determine intent, and make sound re-engineering decisions.
- Clear communication skills and the ability to work effectively with technical and non-technical stakeholders.
Preferred Skills
- Experience with Azure Data Lake Storage Gen2 and Oracle GoldenGate for real-time integration.
- Exposure to Azure Analysis Services and Power BI consumption-layer patterns.
- Experience implementing governance features in Databricks, including Unity Catalog and audit logging.
- Familiarity with Azure DevOps, GitHub Actions, Terraform, ARM, or Docker.
- Knowledge of automated testing approaches for data pipelines, including reconciliation-based validation.
- Experience integrating sources such as Oracle, SQL Server, Azure SQL Database, file systems, Salesforce, or APIs.
- Exposure to Azure Purview, monitoring tools such as Dynatrace, or streaming technologies such as Event Hub, Kafka, or Kinesis.
- Background in regulated or enterprise-scale environments.
Horizontal is committed to building an inclusive workplace where different perspectives, experiences, and approaches are valued. We welcome candidates who share our commitment to diversity, equity, and inclusion and who are excited to contribute in a collaborative, respectful environment.
By applying for this position, you acknowledge and agree that Horizontal Talent may contact you regarding your application using automated technology, including phone calls, SMS/text messages, or email, which may be delivered by our virtual AI recruiter, Alex.
Please apply through this online posting or by visiting our Job Board at Applications will be accepted for 4 weeks. For those that join the team, we offer competitive compensation and benefits including medical, dental, vision, and retirement. Check out all we have to offer and how you can become part of the Horizontal Talent Team. The pay range for this role is $49 - $85 per hour based on qualifications and experience.
