Senior Data Engineer- SQL Server to Databricks Migration
Job Type: Contract (6-12 Months)- Possible extension
Location: Philadelphia, PA (Hybrid – Flexible Work Model)
Job description:
Job summary
We are seeking a senior data engineer with deep Microsoft SQL Server experience to assess and convert a legacy warehouse to Azure Databricks.
You will read complex T-SQL stored procedures and SSIS packages, document the business rules they contain, rebuild them as Databricks pipelines, and reconcile the results before the legacy platform is retired.
Key responsibilities
• Assess T-SQL stored procedures, SSIS packages and scheduled jobs for complexity, dependencies, and embedded business rules.
• Recommend, for each object, whether to retire, convert or redesign it.
• Convert legacy logic to Databricks SQL and PySpark pipelines using agreed patterns and conversion tooling.
• Review and correct automatically converted code.
• Build and run reconciliation between SQL Server and Databricks and resolve differences with business analysts.
• Migrate job schedules and dependencies to Databricks workflows.
• Support cutover and the evidence needed to decommission each legacy component.
• Document data flows, mappings and design decisions in the client's repositories.
Required qualifications
• 8+ years in data engineering or database development, including 5+ years of hands-on T-SQL and SSIS.
• 2+ years building pipelines on Databricks, or an equivalent Spark-based lakehouse platform, with Delta Lake.
• At least one completed migration from SQL Server or another warehouse to a cloud data platform.
• Strong SQL performance tuning and the ability to reverse-engineer undocumented code.
• Experience with data reconciliation and testing at scale.
• Git-based development with pull requests and automated deployment.
Preferred qualifications
• Databricks Lakeflow pipelines and jobs, and Unity Catalog.
• Code-conversion tools such as Databricks Lakebridge.
• Databricks Data Engineer Associate or Professional certification.
• Financial services or wealth management data.
• Experience with enterprise schedulers such as SQL Agent or ActiveBatch.