Role - Senior Data Engineer
Location - Remote, Minneapolis MN
Experience: 4 to 6 Years (Data Engineering) with Large-Scale Cloud Data Platform Migrations
Role Overview
We are seeking experienced Data Engineers to lead and support the migration of large-scale analytical and reporting workloads from Amazon Redshift to Databricks Lakehouse Platform. The ideal candidate will have hands-on experience in data warehouse modernization, ETL/ELT migration, performance optimization, cloud-native architectures, and big data processing.
Key Responsibilities
• Lead the migration of data pipelines, data models, and reporting workloads from Redshift to Databricks.
• Perform assessment and discovery of existing Redshift environments, schemas, workloads, and dependencies.
• Design and implement scalable data solutions using Databricks, Delta Lake, and Spark.
• Convert SQL-based ETL/ELT processes into Databricks notebooks, workflows, and pipelines.
• Optimize data ingestion, transformation, and storage for performance and cost efficiency.
• Develop batch and streaming data pipelines using PySpark and Spark SQL.
• Implement data quality, reconciliation, and validation frameworks during migration.
• Collaborate with business stakeholders, architects, and analytics teams to ensure successful migration and adoption.
• Troubleshoot performance bottlenecks and tune Spark jobs, Delta tables, and SQL workloads.
• Establish CI/CD processes and infrastructure-as-code practices for Databricks environments.
• Document migration strategies, architecture, and operational procedures.
Required Skills
Data Engineering
• Strong experience with large-scale data warehousing and data lake architectures.
• Hands-on experience with Amazon Redshift and Databricks.
• Expertise in PySpark, Spark SQL, and Python.
• Strong SQL development and query optimization skills.
• Experience with data modeling (Star Schema, Snowflake Schema, Dimensional Modeling).
Migration Experience
• Prior experience migrating from:
o Redshift to Databricks
o Teradata/Snowflake/Hadoop to Databricks (preferred)
• Data validation, reconciliation, and cutover planning.
• Performance tuning and workload modernization.
compliant working and operation of supported GxP systems
