Lead Data Engineer

Overview

On Site
Accepts corp to corp applications
Contract - C2C,C2H

Skills

sql
hadoop
.net
ETL
Kafka
azure
spark
Pyspark
Databricks

Job Details

Job Title: Lead Data Engineer
Location: Irving, TX - Hybrid

Must have:

  • Expert in Databricks, PySpark and Azure data services like ADF, Synapse etc.
  • Must have lead level experience
  • 8+ years of professional data engineering experience, with proven technical leadership.
  • Deep understanding of Azure tech stack (Databricks, ADF, Key Vault, ASQL, ADLS Gen2, Service Principles etc.)
  • Deep experience with Databricks (design, optimization, notebooks, stored procedures, UDFs, etc).
  • Hands-on experience with ETL tools such as SSIS and Azure Data Factory.
  • Experience working with API calls and Ingesting data from external source systems
  • Experience with SQL, Python and PySpark
  • Experience with building dynamic and scalable solutions for data ingestion and curation
  • Experience in data engineering and MS SQL Server and T-SQL development.
  • Strong understanding of data lake/delta lake architectures and modern data pipelines.
  • Strong understanding of medallion architectures and design patterns.
  • Strong understanding of data modeling techniques.
  • Experience handling large-scale data volumes (millions of records).
  • Proficiency in building and maintaining CI/CD pipelines.
  • Experience with IaC technologies such as Terraform or Pulumi.
  • Exposure to big data tools (Spark, Hadoop, Kafka) preferred.
  • Experience with reporting tools such as Power BI or Tableau a plus.
  • Knowledge of .NET development beneficial for cross-team collaboration.
  • Prior experience working in an Agile development environment required.
  • Strong communication and collaboration skills with the ability to translate technical concepts to business partners.

What Success Looks Like:

  • You're the go-to technical lead for all things related to data design, integration, and performance.
  • You enable consistency and best practices across Azure, Databricks, SQL, ETL, and data engineering domains.
  • You mentor others while remaining hands-on with architecture and solution delivery.
  • You ensure the Internal Analytics platform continues to evolve as a reliable, scalable data source for the organization.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.