Data Engineer (Need Lead )

Overview

Hybrid
Depends on Experience
Contract - W2
Contract - 12 Month(s)
100% Travel

Skills

Azure Databricks
ETL/ELT
Apache Spark
Delta Lake
Azure
Azure SQL

Job Details

  • Design and implement scalable data pipelines and architectures on Azure Databricks.
  • Optimize ETL/ELT workflows, ensuring efficiency in data processing, storage, and retrieval.
  • Leverage Apache Spark, Delta Lake, and Azure-native services to build high-performance data solutions.
  • Ensure best practices in data governance, security, and compliance within Azure environments.
  • Troubleshoot and fine-tune Spark jobs for optimal performance and cost efficiency.
  • Lead the migration of Azure SQL to Azure Databricks, ensuring a seamless transition of data workloads.
  • Design and implement scalable data pipelines to extract, transform, and load (ETL/ELT) data from Azure SQL into Databricks Delta Lake.
  • Optimize Azure SQL queries and indexing strategies before migration to enhance performance in Databricks.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.