Role Overview
We are seeking an experienced AWS Migration Platform Engineer to design, develop, and operate a secure, scalable, and highly reliable migration platform for transferring large volumes of object data from OBSv2 to Amazon S3.
The role focuses on building automated migration workflows, orchestrating AWS services, managing high-volume data transfers, and implementing robust governance, monitoring, security, and operational controls. The engineer will be responsible for developing production-ready migration capabilities that support both manifest-based and dynamic migration approaches across enterprise environments.
The ideal candidate will have strong hands-on experience with AWS serverless services, containerized workloads, Python, DynamoDB, rclone, networking, and large-scale object storage migrations.
Key Responsibilities
· Design, develop, and maintain end-to-end migration workflows to transfer object data from OBSv2 source buckets to Amazon S3 destination buckets.
· Build and operate AWS-based orchestration solutions using Step Functions, Lambda, ECS Fargate, AWS Batch, DynamoDB, SQS, and rclone.
· Implement manifest-based and dynamic migration strategies, including object discovery, prefix-based partitioning, batching, sharding, task sizing, retries, and recovery mechanisms.
· Develop centralized governance capabilities for bandwidth allocation, queue management, job prioritization, throttling, and migration progress tracking.
· Build reusable, containerized rclone migration components supporting high-volume data transfers, checksum validation, dynamic bandwidth control, and progress reporting.
· Design DynamoDB-backed state management for migration jobs, tasks, queues, bandwidth allocation, and completion summaries.
· Implement comprehensive observability using Amazon CloudWatch dashboards, logs, metrics, alarms, health checks, and operational reports.
· Configure secure multi-account AWS deployments using IAM roles, cross-account access, STS assume-role, Secrets Manager, and least-privilege security principles.
· Optimize migration performance by balancing object volumes, transfer concurrency, compute capacity, network throughput, and bandwidth constraints.
· Collaborate with infrastructure, security, networking, and data engineering teams to resolve dependencies and ensure migration readiness.
· Develop CI/CD pipelines, deployment automation, environment-specific configurations, runbooks, and operational documentation.
· Support integration testing, performance testing, production cutover, issue resolution, restart procedures, and post-migration validation.
· Ensure reliable production operations through proactive monitoring, failure handling, recovery planning, and stakeholder reporting.
Mandatory Technical Skills
AWS Serverless and Orchestration
· Strong hands-on experience with AWS Lambda, Step Functions, SQS, SNS, EventBridge, and API Gateway.
· Experience with AWS Systems Manager (SSM) Parameter Store and Secrets Manager.
· Ability to design fault-tolerant, event-driven workflows with retry and error-handling mechanisms.
AWS Compute and Containers
· Strong working knowledge of Amazon ECS Fargate and AWS Batch.
· Experience with container images, task definitions, sidecar patterns, resource sizing, and runtime configuration.
· Ability to optimize containerized workloads for large-scale data transfer operations.
Amazon S3 and Object Storage
· Deep understanding of Amazon S3, S3 Intelligent-Tiering, S3-compatible APIs, object metadata, prefixes, and lifecycle management.
· Experience handling large object transfers, high object counts, and enterprise-scale storage environments.
· Understanding of S3 storage optimization and destination validation.
Object Storage Migration and rclone
· Hands-on experience migrating large object datasets from on-premises or S3-compatible object storage platforms to Amazon S3.
· Experience with Dell ECS, OBSv2, or similar enterprise object storage systems is highly desirable.
· Strong experience configuring and operating rclone for high-volume transfers, including concurrency, transfers, checkers, retries, checksum validation, no-traverse, statistics, logging, and bandwidth limiting.
Python Development
· Strong Python programming skills for developing AWS Lambda functions, Fargate services, and migration automation utilities.
· Experience with boto3, JSON processing, exception handling, structured logging, dataclasses, streaming files, and API integrations.
· Ability to develop reusable, maintainable, and production-ready automation components.
DynamoDB and State Management
· Experience designing and operating DynamoDB tables, partition and sort keys, indexes, updates, scans, and batch writes.
· Understanding of Decimal handling, job tracking, task orchestration, queue state, and distributed workflow state management.
AWS Networking
· Strong understanding of AWS Direct Connect, Transit Gateway, VPCs, private subnets, routing, S3 Gateway Endpoints, and NAT Gateway configurations.
· Experience designing high-throughput network paths for large-scale data transfers.
· Ability to troubleshoot network connectivity, routing, and performance bottlenecks.
IAM and Security
· Strong understanding of IAM policies, roles, cross-account access, STS assume-role, API keys, and Secrets Manager.
· Experience implementing least-privilege access, secure credential management, and privacy safeguards.
· Ability to support secure deployment and operations across multiple AWS accounts.
Monitoring and Observability
· Hands-on experience with CloudWatch metrics, logs, alarms, dashboards, and structured logging.
· Ability to implement operational telemetry, migration progress reporting, health checks, and failure alerts.
· Experience developing monitoring and troubleshooting procedures for production workloads.
CI/CD and Deployment Automation
· Experience with GitLab repositories, CI/CD pipelines, and automated infrastructure deployment.
· Ability to manage environment-specific configuration, deployment validation, release controls, and safe resource teardown.
· Familiarity with version control, code reviews, and production release practices.
Domain Knowledge and Required Expertise
· Understanding of large-scale object storage migration patterns, including manifest-based migration, dynamic object discovery, prefix partitioning, batching, and validation.
· Ability to balance migration throughput against data volume, object count, compute resources, network capacity, and rclone concurrency.
· Knowledge of bandwidth allocation, dynamic throttling, queue processing, job prioritization, retries, and escalation mechanisms.
· Strong operational discipline covering monitoring, runbooks, restart and recovery procedures, completion checks, and stakeholder communications.
· Ability to create clear technical documentation, architecture diagrams, input/output contracts, implementation notes, and support procedures.
· Experience supporting enterprise-scale migration programs in production environments, preferably with security and compliance requirements.
Expected Deliverables
· A production-ready migration orchestration platform supporting both manifest-based and dynamic migration workflows.
· Centralized governance for bandwidth allocation, queue processing, throttling, job prioritization, and completion reporting.
· A reusable rclone-based migration runtime with progress reporting, checksum validation, and dynamic bandwidth control.
· DynamoDB-backed state management for migration jobs, tasks, queues, bandwidth allocation, and execution summaries.
· CloudWatch dashboards, alarms, logs, health checks, and operational telemetry.
· Automated deployment pipelines, environment-specific configurations, operational runbooks, and technical documentation.
· A secure, scalable, and recoverable platform ready for enterprise production migration and cutover.
Preferred Qualifications
· Experience with Dell ECS, OBSv2, or other S3-compatible enterprise object storage platforms.
· Experience delivering large-scale enterprise data migrations in regulated or business-critical production environments.
· Experience with hub-and-spoke architecture and centralized orchestration across multiple AWS accounts.
· AWS certifications such as AWS Certified Solutions Architect or AWS Certified Developer.
· Experience with infrastructure as code and automated cloud resource provisioning.
· Strong technical communication skills and the ability to provide clear, leadership-ready status reports.
