Data Engineer

Data Migration - Data Engineer JD

Alpharetta, GA

Fulltime

$110- $130K Per Annum

Role Overview:

As a Data Engineer focused on this DB2 to SQL Server migration project, you will be responsible for the end-to-end process of extracting, transforming, and loading data, as well as ensuring the integrity and accuracy of the migrated data. You will leverage your deep understanding of both IBM DB2 and SQL Server, advanced SQL skills, and proficiency in Python to build robust and efficient migration pipelines.

Responsibilities:

• Data Migration Strategy and Planning: Participate in the planning and strategizing phases of the data migration project, providing technical insights and recommendations.
• Source Data Analysis: Analyze the structure, content, and quality of data within the IBM DB2 environment to understand migration complexities and potential challenges.
• Schema Mapping and Conversion: Collaborate with database administrators and architects to map DB2 schemas and data types to their SQL Server equivalents, addressing any compatibility issues.
• Data Extraction and Transformation (ETL/ELT): Design, develop, and implement efficient data extraction pipelines from IBM DB2 using various methods, including SQL queries and potentially scripting tools.
• Data Transformation Logic: Develop and implement data transformation logic using Advanced SQL and Python to cleanse, reshape, and prepare data for loading into SQL Server, ensuring data quality and consistency.
• SQL Server Data Loading: Design and implement efficient data loading processes into Microsoft SQL Server, considering performance and scalability.
• Data Validation and Reconciliation: Develop and execute comprehensive data validation strategies and scripts (using SQL and Python) to compare data between the source DB2 and target SQL Server environments, ensuring data accuracy and completeness.
• Performance Optimization: Identify and implement performance optimization techniques for data extraction, transformation, and loading processes.
• Troubleshooting and Issue Resolution: Investigate and resolve data migration issues, including data inconsistencies, errors, and performance bottlenecks.
• Documentation: Create and maintain clear and concise technical documentation for data migration processes, scripts, and validation procedures.
• Collaboration: Work closely with database administrators, application developers, and project managers throughout the migration lifecycle.
• Adherence to Best Practices: Follow data migration best practices and ensure the security and integrity of the data throughout the process.

Required Skills:

• Advanced SQL Proficiency: Deep understanding of SQL syntax, query optimization, stored procedures, functions, and performance tuning in both IBM DB2 and Microsoft SQL Server.
• Strong Python Programming Skills: Proficiency in Python for data manipulation, scripting, automation, and building data migration and validation pipelines. Experience with relevant libraries (e.g., pandas, pyodbc).
• Solid Understanding of IBM DB2: Experience working with IBM DB2 databases, including data structures, system catalogs, and common utilities.
• Solid Understanding of Microsoft SQL Server: Experience working with Microsoft SQL Server databases, including data structures, system objects, and common tools.
• Proven Data Migration Experience: Demonstrated experience in successfully migrating data between different database systems, preferably involving IBM DB2 and SQL Server.
• Data Validation Expertise: Strong experience in designing and implementing data validation strategies, writing validation scripts, and identifying and resolving data discrepancies.
• Understanding of Data Integration Concepts: Familiarity with ETL/ELT principles and best practices.
• Strong Problem-Solving and Analytical Skills: Ability to analyze complex data issues, identify root causes, and implement effective solutions.
• Excellent Communication and Collaboration Skills: Ability to communicate technical concepts clearly and work effectively within a team.
• Strong Documentation Skills: Ability to create clear and comprehensive technical documentation.

Preferred Skills:

• Experience with database migration tools (e.g., SSMA, other third-party tools).
• Familiarity with data modeling concepts and techniques.
• Experience with data quality tools and processes.
• Knowledge of scripting languages beyond Python (e.g., Shell scripting).
• Experience with Agile methodologies.