Databricks Engineer

  • New York, NY
  • Posted 19 hours ago | Updated 19 hours ago

Overview

Hybrid
Depends on Experience
Contract - W2

Skills

Databricks
ETL

Job Details

  • Expertise in designing and developing ETL code to integrate various data sources.
  • Proficiency in creating efficient data pipelines on the Databricks platform.
  • Experience in developing, maintaining, and optimizing ETL pipelines.
  • Ability to handle both streaming and batch data processing.
  • Provide comprehensive support for ETL processes from data ingestion to final output.
  • Ensure data integrity and consistency throughout the ETL lifecycle.
  • Write and execute unit tests and integration test cases for ETL code.
  • Ensure high-quality code through rigorous testing practices.
  • Implement monitoring solutions for ETL jobs to ensure timely and successful data processing.
  • Proactively identify and resolve issues in ETL workflows.
  • Optimize ETL jobs for maximum performance and efficiency.
  • Conduct performance tuning and troubleshooting of ETL processes.
  • Minimum 10 years of experience in this field

Preferred Skills

  • In-depth knowledge of Databricks platform and technologies Delta Lake, Databricks SQL, and Databricks Workflows.
  • Experience with Azure cloud platforms
  • Familiarity with Azure Data Lake cloud storage cloud storage
  • Knowledge of data warehousing, data modeling, and best practices.
  • Proficiency in programming languages such as Python, SQL, Scala, or R.
  • Experience with big data technologies such as Apache Spark, Hadoop, or Kafka.
  • Knowledge of working with Hadoop for migration projects from Hadoop to Databricks
  • Familiarity with DevOps practices and tools such as CI/CD, Git, etc.
  • Knowledge of infrastructure as code (IaC) tools like Terraform
  • Experience with implementing data governance and security measures in a cloud environment.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.