Role: Senior Azure Data Migration & Dataverse Integration Engineer
Company: Arch Systems
Client: US Federal Govt.
Location: Remote
Type: Full-time
Position Summary
Seeking a Senior Azure Data Migration & Dataverse Integration Engineer to design, develop, implement, and support enterprise data migration and synchronization capabilities between Microsoft SQL Server and Microsoft Dataverse.
The position requires a senior hands-on engineer capable of owning the complete data pipeline lifecycle, including source analysis, data mapping, extraction, transformation, migration, synchronization, validation, exception handling, monitoring, performance optimization, and automated deployment.
The engineer will develop and maintain repeatable pipelines supporting initial bulk migration, incremental migration, Change Data Capture (CDC), ongoing synchronization, and transition/cutover activities between legacy SQL Server applications and modern Microsoft Power Platform/Dataverse solutions.
Primary Responsibilities
The Data Migration Engineer will:
Design and implement end-to-end SQL Server-to-Dataverse data migration pipelines.
Develop automated pipelines using Azure Data Factory (ADF) and supporting Azure services.
Analyze legacy SQL Server databases, tables, views, relationships, stored procedures, and data dependencies.
Map legacy SQL Server data structures to Dataverse tables, columns, relationships, choices, lookups, and alternate keys.
Develop repeatable processes for:
Initial/bulk data migration
Incremental data migration
Change Data Capture
Delta processing
Scheduled synchronization
Near-real-time synchronization
Legacy-to-Dataverse synchronization
Dataverse-to-legacy synchronization where required
Cutover and final synchronization
Develop and maintain ADF pipelines, datasets, linked services, integration runtimes, triggers, parameters, variables, and reusable pipeline components.
Develop transformation and data-cleansing logic required to convert legacy data into Dataverse-compatible structures.
Implement controlled sequencing to maintain parent/child relationships and referential integrity.
Develop lookup resolution and cross-reference processes for Dataverse GUIDs and legacy SQL identifiers.
Implement restartable and recoverable migration processes.
Develop pipeline logging, audit, exception handling, retry, and reconciliation capabilities.
Validate migrated data for completeness, accuracy, integrity, and consistency.
Perform source-to-target record-count and field-level reconciliation.
Troubleshoot failed records, pipeline failures, API limitations, data-quality issues, and performance problems.
Develop automated deployment processes for data pipeline components across DEV, TEST, UAT, and PROD environments.
Azure Data Factory / Pipeline Responsibilities
The candidate must have advanced hands-on experience designing and developing Azure Data Factory solutions.
Responsibilities include:
ADF pipeline architecture and development
Copy Activities
Lookup Activities
Stored Procedure Activities
ForEach and Until processing
Conditional processing
Pipeline dependencies
Parameterized pipelines
Dynamic expressions
Metadata-driven pipeline development
Linked Services
Datasets
Integration Runtime configuration
Managed Identity authentication
Azure Key Vault integration
Pipeline triggers and scheduling
Incremental load processing
Watermark processing
CDC patterns
Retry and failure handling
Pipeline logging and auditing
Performance tuning and parallelization
Environment-specific configuration
Infrastructure and pipeline deployment automation
The engineer should be capable of developing reusable, metadata-driven pipeline patterns rather than creating independent hard-coded pipelines for every table.
SQL Server Responsibilities
Strong SQL Server development and data engineering skills are required, including:
Complex T-SQL development
Stored procedures
Views
Common Table Expressions
Temporary and staging tables
Data profiling
Data cleansing
Data transformation
Joins and relationship analysis
Primary and foreign key analysis
Indexing and query optimization
SQL Server Change Data Capture
Change Tracking
Watermark-based incremental extraction
Data validation queries
Reconciliation queries
Duplicate detection
Referential-integrity validation
The engineer must be capable of analyzing legacy databases with limited documentation and determining the technical relationships necessary to migrate data successfully.
Microsoft Dataverse Responsibilities
Advanced Dataverse knowledge is required, including:
Dataverse tables and columns
Standard and custom tables
Table relationships
One-to-many and many-to-many relationships
Lookups
Choice columns
Alternate keys
GUID management
Ownership and security considerations
Dataverse Web API
Dataverse APIs and connectors
Upsert patterns
Batch processing
Duplicate detection
Dataverse service protection/API limits
Application users and service principals
Environment configuration
Solutions and ALM
The candidate must understand how Dataverse differs from a traditional relational SQL database and design migration processes accordingly.
Data Synchronization
The engineer will design synchronization processes supporting coexistence between legacy and modernized applications during phased transition.
This includes:
SQL Server → Dataverse synchronization
Dataverse → SQL Server synchronization where required
Unidirectional and bidirectional synchronization patterns
Incremental synchronization
CDC and watermark processing
Event-driven integration where applicable
Conflict detection and resolution
Source-of-record determination
Duplicate prevention
Idempotent processing
Failed-record reprocessing
Synchronization audit history
Data latency monitoring
Reconciliation between source and target systems
The engineer must understand the risks associated with bidirectional synchronization, including update conflicts, synchronization loops, transaction ordering, and determination of the authoritative system of record.
Data Migration Architecture
The candidate should be capable of implementing a controlled migration architecture such as:
Legacy SQL Server → Extraction → Staging/Bronze → Validation/Transformation → Silver → Business Rules/Mapping → Gold/Target-Ready Data → ADF Load Pipeline → Dataverse → Reconciliation
The architecture must support traceability from the original source record through transformation and final Dataverse record creation.
Pipeline Monitoring and Operations
Develop operational capabilities to monitor migration and synchronization processing, including:
Pipeline execution status
Records extracted
Records transformed
Records successfully loaded
Records rejected
Warning and error counts
Source and target record counts
Processing duration
Synchronization latency
Retry status
Failed-record queues
Reconciliation status
Implement monitoring using applicable Microsoft technologies such as:
DevOps and CI/CD
The engineer will establish controlled deployment and configuration management for all data migration components.
Responsibilities include:
Azure DevOps Repos
Branching and source-control practices
Pull requests and code reviews
Azure DevOps Pipelines
Automated DEV/TEST/UAT/PROD deployments
Environment-specific parameters
Infrastructure as Code
Bicep and/or ARM templates
ADF deployment automation
SQL database deployment
Dataverse configuration coordination
Key Vault and secret-management integration
Release validation
Rollback procedures
All pipelines, scripts, SQL objects, infrastructure definitions, configuration, and supporting code must be maintained under source control.
Security Requirements
The engineer will implement data migration capabilities consistent with enterprise security requirements, including:
Microsoft Entra ID
Managed Identities
Service Principals
Dataverse Application Users
Role-Based Access Control
Least-privilege access
Azure Key Vault
Secure connection management
Encryption in transit and at rest
Audit logging
Environment separation
Controlled production access
Experience supporting Federal Government, DoD, RMF, NIST 800-53, or comparable regulated environments is preferred.
Required Technical Skills
7+ years of data engineering, database development, integration, or data migration experience.
4+ years of Microsoft Azure data engineering experience.
Advanced Azure Data Factory development experience.
Advanced Microsoft SQL Server and T-SQL experience.
Strong Microsoft Dataverse experience.
Experience migrating enterprise SQL data into Dataverse.
Experience with CDC, incremental loads, delta processing, and synchronization.
Experience developing ETL/ELT pipelines.
Experience with REST APIs and Dataverse Web API.
Experience with Azure SQL.
Experience with Azure Key Vault and Managed Identities.
Experience with Azure DevOps source control and pipelines.
Experience implementing automated data validation and reconciliation.
Preferred Technical Experience
Experience with several of the following is highly desirable:
Preferred Certifications
One or more of the following Microsoft certifications is preferred:
DP-203 – Azure Data Engineer Associate or current Microsoft equivalent
PL-400 – Power Platform Developer Associate
PL-600 – Power Platform Solution Architect Expert
Microsoft Azure Administrator, Developer, or Solutions Architect certification
Desired Candidate Profile
This position requires a hands-on senior data engineer, not solely a data architect or Power Platform administrator.
The successful candidate must be able to independently take a legacy SQL data requirement through the complete technical lifecycle:
Legacy SQL Analysis → Data Profiling → Source-to-Target Mapping → Pipeline Design → ADF Development → Transformation → Dataverse Load → CDC/Synchronization → Validation → Reconciliation → CI/CD → Production Operations
The engineer must be capable of working directly with application developers, Power Platform developers, database administrators, solution architects, business analysts, and system owners to determine how legacy data will be migrated, synchronized, validated, and ultimately transitioned from the legacy environment.
Key Deliverables
The position will be responsible for producing and maintaining:
Source-to-target data mappings
Data migration pipelines
Data synchronization pipelines
ADF pipeline framework
SQL extraction and transformation scripts
Staging database structures
CDC/incremental processing framework
Dataverse loading processes
Data-quality rules
Exception and retry processes
Migration audit records
Data reconciliation reports
Pipeline monitoring
CI/CD deployment pipelines
Infrastructure-as-Code definitions
Migration runbooks
Cutover procedures
Production support documentation
Technical architecture and data-flow diagrams
Never miss an opportunity! Create an alert based on the job you applied for.
