Senior Azure Data Migration & Dataverse Integration Engineer

Role: Senior Azure Data Migration & Dataverse Integration Engineer
Company: Arch Systems
Client: US Federal Govt.
Location: Remote
Type: Full-time

Position Summary

Seeking a Senior Azure Data Migration & Dataverse Integration Engineer to design, develop, implement, and support enterprise data migration and synchronization capabilities between Microsoft SQL Server and Microsoft Dataverse.

The position requires a senior hands-on engineer capable of owning the complete data pipeline lifecycle, including source analysis, data mapping, extraction, transformation, migration, synchronization, validation, exception handling, monitoring, performance optimization, and automated deployment.

The engineer will develop and maintain repeatable pipelines supporting initial bulk migration, incremental migration, Change Data Capture (CDC), ongoing synchronization, and transition/cutover activities between legacy SQL Server applications and modern Microsoft Power Platform/Dataverse solutions.

Primary Responsibilities

The Data Migration Engineer will:

• Design and implement end-to-end SQL Server-to-Dataverse data migration pipelines.

• Develop automated pipelines using Azure Data Factory (ADF) and supporting Azure services.

• Analyze legacy SQL Server databases, tables, views, relationships, stored procedures, and data dependencies.

• Map legacy SQL Server data structures to Dataverse tables, columns, relationships, choices, lookups, and alternate keys.

• Develop repeatable processes for:

  • Initial/bulk data migration

  • Incremental data migration

  • Change Data Capture

  • Delta processing

  • Scheduled synchronization

  • Near-real-time synchronization

  • Legacy-to-Dataverse synchronization

  • Dataverse-to-legacy synchronization where required

  • Cutover and final synchronization

• Develop and maintain ADF pipelines, datasets, linked services, integration runtimes, triggers, parameters, variables, and reusable pipeline components.

• Develop transformation and data-cleansing logic required to convert legacy data into Dataverse-compatible structures.

• Implement controlled sequencing to maintain parent/child relationships and referential integrity.

• Develop lookup resolution and cross-reference processes for Dataverse GUIDs and legacy SQL identifiers.

• Implement restartable and recoverable migration processes.

• Develop pipeline logging, audit, exception handling, retry, and reconciliation capabilities.

• Validate migrated data for completeness, accuracy, integrity, and consistency.

• Perform source-to-target record-count and field-level reconciliation.

• Troubleshoot failed records, pipeline failures, API limitations, data-quality issues, and performance problems.

• Develop automated deployment processes for data pipeline components across DEV, TEST, UAT, and PROD environments.

Azure Data Factory / Pipeline Responsibilities

The candidate must have advanced hands-on experience designing and developing Azure Data Factory solutions.

Responsibilities include:

• ADF pipeline architecture and development

• Copy Activities

• Lookup Activities

• Stored Procedure Activities

• ForEach and Until processing

• Conditional processing

• Pipeline dependencies

• Parameterized pipelines

• Dynamic expressions

• Metadata-driven pipeline development

• Linked Services

• Datasets

• Integration Runtime configuration

• Managed Identity authentication

• Azure Key Vault integration

• Pipeline triggers and scheduling

• Incremental load processing

• Watermark processing

• CDC patterns

• Retry and failure handling

• Pipeline logging and auditing

• Performance tuning and parallelization

• Environment-specific configuration

• Infrastructure and pipeline deployment automation

The engineer should be capable of developing reusable, metadata-driven pipeline patterns rather than creating independent hard-coded pipelines for every table.

SQL Server Responsibilities

Strong SQL Server development and data engineering skills are required, including:

• Complex T-SQL development

• Stored procedures

• Views

• Common Table Expressions

• Temporary and staging tables

• Data profiling

• Data cleansing

• Data transformation

• Joins and relationship analysis

• Primary and foreign key analysis

• Indexing and query optimization

• SQL Server Change Data Capture

• Change Tracking

• Watermark-based incremental extraction

• Data validation queries

• Reconciliation queries

• Duplicate detection

• Referential-integrity validation

The engineer must be capable of analyzing legacy databases with limited documentation and determining the technical relationships necessary to migrate data successfully.

Microsoft Dataverse Responsibilities

Advanced Dataverse knowledge is required, including:

• Dataverse tables and columns

• Standard and custom tables

• Table relationships

• One-to-many and many-to-many relationships

• Lookups

• Choice columns

• Alternate keys

• GUID management

• Ownership and security considerations

• Dataverse Web API

• Dataverse APIs and connectors

• Upsert patterns

• Batch processing

• Duplicate detection

• Dataverse service protection/API limits

• Application users and service principals

• Environment configuration

• Solutions and ALM

The candidate must understand how Dataverse differs from a traditional relational SQL database and design migration processes accordingly.

Data Synchronization

The engineer will design synchronization processes supporting coexistence between legacy and modernized applications during phased transition.

This includes:

• SQL Server → Dataverse synchronization

• Dataverse → SQL Server synchronization where required

• Unidirectional and bidirectional synchronization patterns

• Incremental synchronization

• CDC and watermark processing

• Event-driven integration where applicable

• Conflict detection and resolution

• Source-of-record determination

• Duplicate prevention

• Idempotent processing

• Failed-record reprocessing

• Synchronization audit history

• Data latency monitoring

• Reconciliation between source and target systems

The engineer must understand the risks associated with bidirectional synchronization, including update conflicts, synchronization loops, transaction ordering, and determination of the authoritative system of record.

Data Migration Architecture

The candidate should be capable of implementing a controlled migration architecture such as:

Legacy SQL Server → Extraction → Staging/Bronze → Validation/Transformation → Silver → Business Rules/Mapping → Gold/Target-Ready Data → ADF Load Pipeline → Dataverse → Reconciliation

The architecture must support traceability from the original source record through transformation and final Dataverse record creation.

Pipeline Monitoring and Operations

Develop operational capabilities to monitor migration and synchronization processing, including:

• Pipeline execution status

• Records extracted

• Records transformed

• Records successfully loaded

• Records rejected

• Warning and error counts

• Source and target record counts

• Processing duration

• Synchronization latency

• Retry status

• Failed-record queues

• Reconciliation status

Implement monitoring using applicable Microsoft technologies such as:

• Azure Monitor

• Application Insights

• Log Analytics

• ADF monitoring

• Azure SQL logging

• Power BI or operational dashboards

DevOps and CI/CD

The engineer will establish controlled deployment and configuration management for all data migration components.

Responsibilities include:

• Azure DevOps Repos

• Branching and source-control practices

• Pull requests and code reviews

• Azure DevOps Pipelines

• Automated DEV/TEST/UAT/PROD deployments

• Environment-specific parameters

• Infrastructure as Code

• Bicep and/or ARM templates

• ADF deployment automation

• SQL database deployment

• Dataverse configuration coordination

• Key Vault and secret-management integration

• Release validation

• Rollback procedures

All pipelines, scripts, SQL objects, infrastructure definitions, configuration, and supporting code must be maintained under source control.

Security Requirements

The engineer will implement data migration capabilities consistent with enterprise security requirements, including:

• Microsoft Entra ID

• Managed Identities

• Service Principals

• Dataverse Application Users

• Role-Based Access Control

• Least-privilege access

• Azure Key Vault

• Secure connection management

• Encryption in transit and at rest

• Audit logging

• Environment separation

• Controlled production access

Experience supporting Federal Government, DoD, RMF, NIST 800-53, or comparable regulated environments is preferred.

Required Technical Skills

• 7+ years of data engineering, database development, integration, or data migration experience.

• 4+ years of Microsoft Azure data engineering experience.

• Advanced Azure Data Factory development experience.

• Advanced Microsoft SQL Server and T-SQL experience.

• Strong Microsoft Dataverse experience.

• Experience migrating enterprise SQL data into Dataverse.

• Experience with CDC, incremental loads, delta processing, and synchronization.

• Experience developing ETL/ELT pipelines.

• Experience with REST APIs and Dataverse Web API.

• Experience with Azure SQL.

• Experience with Azure Key Vault and Managed Identities.

• Experience with Azure DevOps source control and pipelines.

• Experience implementing automated data validation and reconciliation.

Preferred Technical Experience

Experience with several of the following is highly desirable:

• Azure Data Lake Storage Gen2

• Azure SQL Database

• Azure SQL Managed Instance

• Microsoft Fabric

• Azure Functions

• Azure Service Bus

• Azure Event Grid

• Logic Apps

• Power Automate

• Power Apps

• Power BI

• Synapse Analytics

• C#/.NET

• Python

• PowerShell

• Bicep

• Git

• Azure DevOps

Preferred Certifications

One or more of the following Microsoft certifications is preferred:

• DP-203 – Azure Data Engineer Associate or current Microsoft equivalent

• PL-400 – Power Platform Developer Associate

• PL-600 – Power Platform Solution Architect Expert

• Microsoft Azure Administrator, Developer, or Solutions Architect certification

Desired Candidate Profile

This position requires a hands-on senior data engineer, not solely a data architect or Power Platform administrator.

The successful candidate must be able to independently take a legacy SQL data requirement through the complete technical lifecycle:

Legacy SQL Analysis → Data Profiling → Source-to-Target Mapping → Pipeline Design → ADF Development → Transformation → Dataverse Load → CDC/Synchronization → Validation → Reconciliation → CI/CD → Production Operations

The engineer must be capable of working directly with application developers, Power Platform developers, database administrators, solution architects, business analysts, and system owners to determine how legacy data will be migrated, synchronized, validated, and ultimately transitioned from the legacy environment.

Key Deliverables

The position will be responsible for producing and maintaining:

• Source-to-target data mappings

• Data migration pipelines

• Data synchronization pipelines

• ADF pipeline framework

• SQL extraction and transformation scripts

• Staging database structures

• CDC/incremental processing framework

• Dataverse loading processes

• Data-quality rules

• Exception and retry processes

• Migration audit records

• Data reconciliation reports

• Pipeline monitoring

• CI/CD deployment pipelines

• Infrastructure-as-Code definitions

• Migration runbooks

• Cutover procedures

• Production support documentation

• Technical architecture and data-flow diagrams