ADF Data Engineer

Overview

Remote
$120,000 - $140,000
Full Time

Skills

Azure
ADF
SQL
ETL
Data Lake

Job Details

ADF Data Engineer Lead

Chicago, IL

Hybrid/Remote

Key Responsibilities

Design and Development:

  • Develop, implement, and optimize scalable data pipelines and workflows in Azure Data Factory (ADF).
  • Design and implement ETL/ELT processes to ingest, transform, and load data from various sources into Azure Data Lake, Azure SQL Database, or other storage solutions.
  • Collaborate with data architects to design data models and ensure alignment with business requirements.

Data Integration:

  • Integrate data from structured, semi-structured, and unstructured data sources such as APIs, databases, flat files, and cloud storage.
  • Implement data transformation logic using ADF Mapping Data Flows, Databricks, or other Azure services.

Performance Optimization:

  • Monitor and optimize data pipelines to ensure high performance, reliability, and scalability.
  • Troubleshoot and resolve issues related to data ingestion, transformation, and delivery.

Collaboration:

  • Work closely with cross-functional teams, including data analysts, business stakeholders, and software engineers, to understand data requirements and deliver solutions.
  • Provide technical guidance and mentorship to junior data engineers.

Governance and Security:

  • Implement data governance best practices, including data lineage, metadata management, and data quality checks.
  • Ensure compliance with security and privacy standards, including role-based access control (RBAC) and encryption.

Documentation:

  • Create and maintain detailed technical documentation for data pipelines, workflows, and processes.
  • Develop runbooks and operational guides for production support.

Required Skills and Qualifications

Technical Skills

  • Strong hands-on experience with Azure Data Factory (ADF), including pipeline orchestration and Mapping Data Flows.
  • Proficiency in Azure Synapse Analytics, Azure Data Lake, Azure SQL Database, and Azure Blob Storage.
  • Solid understanding of ETL/ELT processes and data integration best practices.
  • Experience with programming languages such as Python, SQL, or C#.

Familiarity with Azure Databricks, Azure Functions, and Logic Apps is a plus

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.