Lead Data Engineer

  • Milwaukee, WI
  • Posted 4 days ago | Updated 5 hours ago

Overview

Hybrid
Depends on Experience
Contract - W2
Contract - 6 Month(s)
No Travel Required

Skills

ADF
Data Lake
Analytics
Data Engineering
SQL
Data Warehouse

Job Details

Position Title: Lead Data Engineer

Duration: 3-6 months+

Location: Milwaukee, WI, 2-3 days onsite/weekly

Overview:

We are seeking a Data Engineer with strong hands-on expertise in designing and implementing scalable data ingestion, processing, governance, and ML integration solutions on Azure. The role will focus on Azure Data Explorer (ADX), Azure Data Factory (ADF), and machine learning pipeline integration with Azure ML Studio.

Key Skills (Required)

Azure Data Explorer (ADX)

  • Proven experience setting up ADX as a data warehouse for telemetry and operational data.
  • Strong knowledge of data modeling and schema design for telemetry and time-series data.
  • Strong knowledge of Kusto Query Language (KQL) for querying, aggregation, and transformations.
  • Designing and implementing ingestion strategies for streaming ingestion (Event Hub, Stream Analytics).
  • Building batch ingestion flows into ADX.
  • Implementing continuous data export from ADX to Azure Data Lake.
  • Experience implementing data quality frameworks, anomaly detection, and governance practices within ADX.

Azure Data Factory (ADF)

  • Building end-to-end pipelines moving data from SQL databases to ADX and Data Lake.
  • Designing pipelines that are scalable, reliable, and monitored.
  • Implementing robust error handling, retry, and alerting mechanisms.
  • Troubleshooting and optimizing pipelines for performance and cost efficiency.
  • Experience with DevOps for Data Engineering (CI/CD for ADF pipelines, ADX ingestion).
  • Experience implementing data quality frameworks and governance practices across ADF pipelines.

Machine Learning Integration

  • Designing and building data pipelines that feed curated data into ML models deployed as endpoints in Azure ML Studio.
  • Automating real-time and batch scoring workflows using ML endpoints.
  • Enabling feature delivery pipelines for ML training and inference.
  • Experience with DevOps for ML data pipelines (automated deployment, monitoring, CI/CD).

Experience implementing data quality validation and governance practices for ML data flows

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.