Sr. Databricks Engineer

Overview

Remote
$50 - $70
Contract - W2
Contract - Independent
Able to Provide Sponsorship

Skills

Apache Spark
Cloud Computing
Cloud Storage
Computer Networking
Conflict Resolution
Continuous Delivery
Continuous Improvement
Continuous Integration
Data Architecture
Data Engineering
Data Processing
Database
Databricks
IT Management
Machine Learning (ML)
Management
Microsoft Azure
Network
NoSQL
Orchestration
Problem Solving
Scalability
Scripting
Streaming
Terraform
Video
Virtual Private Cloud
Workflow

Job Details

Job Title: Sr. Databricks Engineer


Location: Remote (Quarterly travel to Boston or NYC)
Duration: Through March 2026, with annual renewal (Long-term opportunity)
Interview Mode: Video Interview
Visa Restrictions: Open to all visa types
Pay Rate: $80/hr C2C

Job Description

We are urgently seeking a Senior Databricks Engineer for a long-term engagement with one of our premier clients. This is a remote position with quarterly travel to either the Boston or NYC office.

As a key member of the Data Engineering team, you will be responsible for architecting, deploying, and optimizing Databricks environments within Azure, and guiding cross-functional teams on best practices for building scalable and secure data pipelines.

Key Responsibilities:

  • Architect, configure, and deploy Databricks workspaces in Azure.

  • Design secure and high-performance connections to a variety of data sources including on-premise systems, cloud storage, relational/NoSQL databases, and APIs.

  • Build and maintain data pipelines using Delta Live Tables (DLT), Apache Spark, and other modern data stack tools.

  • Work closely with security teams to ensure alignment with enterprise governance, IAM policies, and network requirements (VPC, Private Link, etc.).

  • Optimize Databricks cluster configurations for cost-efficiency, scalability, and performance.

  • Provide technical leadership to engineers and analysts on best practices in data ingestion, transformation, and ML workflows within Databricks.

  • Troubleshoot complex issues related to performance, connectivity, and platform configurations.

  • Develop infrastructure-as-code and automation scripts using tools such as Terraform, CI/CD pipelines, etc.

  • Keep up-to-date with Databricks product releases, industry innovations, and advocate for continual process improvement.

Required Skills & Qualifications:

  • Proven experience with Databricks on Azure, including workspace setup and security integration.

  • Deep expertise in Apache Spark, Delta Lake, and streaming/batch data processing.

  • Strong understanding of data architecture, pipeline orchestration, and cloud-native services.

  • Hands-on experience with Terraform, CI/CD automation, and infrastructure management.

  • Solid knowledge of IAM, networking (VPC, Private Link), and security best practices in cloud environments.

  • Excellent problem-solving skills and the ability to work independently in a fast-paced, remote setting.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.