Azure Data Engineer

ADF Consultant

Remote

Full Time

Top Required skills: Azure Certified and Azure data factory, ETL

Deep expertise in data integration/ETL/ELT and pipeline orchestration:

• ADF s core role is to ingest, transform, and move data across disparate sources. A consultant must know how to design robust, efficient, and maintainable pipelines (copy activities, mapping data flows, data transformations)
• You should understand incremental vs full load patterns, schema drift, partitioning, change data capture, and performance tuning of data flows
• You should also be comfortable orchestrating dependencies, scheduling, error handling, retries, monitoring, and alerting across activities.

Solid proficiency in SQL, scripting, and data transformation logic:

• Most transformations, aggregations, joins, filters, deduplication, and data cleansing will rely on SQL skills
• In many cases, you ll also need scripting (e.g. Python, PowerShell, or other language) to handle data preparations, custom logic, or pre/post-processing steps
• Familiarity with handling structured, semi-structured (JSON, XML), and unstructured data is also valuable

Familiarity with the broader Azure data ecosystem & architectural thinking:

• ADF rarely works in isolation you ll integrate with Azure Data Lake Storage (ADLS), Azure SQL Database, Azure Synapse Analytics, Databricks, Cosmos DB, etc. You need to understand when and how to use each.
• You must know best practices in data architecture: data lakes vs data warehouses, staging layers, medallion (bronze/silver/gold) architecture, and how ADF fits into that.
• Understanding security, scaling, governance, and cost optimization in Azure is crucial (e.g. secure data movement, encryption, role-based access controls, integration runtimes, network design)