Apply Now

W2 ONLY -Data Engineer ( Databricks, Scala , PYSPARK)

Dallas, TX, US • Posted 2 days ago • Updated 2 days ago

Contract W2

12 Months

No Travel Required

On-site

Depends on Experience

cloudingest inc

Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Job Details

Skills

Apache Spark
Databricks
Scala
PySpark

Summary

Experienced Senior Data Engineer to work on a focused migration project — converting existing Scala-based Spark pipelines to PySpark and transitioning job workloads from classic Databricks clusters to Databricks Serverless compute. The ideal candidate has hands-on experience with both Scala and PySpark, deep knowledge of Azure Databricks, and a strong understanding of Serverless compute constraints and best practices.

Key Responsibilities

Review and analyze existing Scala Spark jobs — identify dependencies, performance bottlenecks, and Serverless compatibility issues.
Convert Scala pipelines to PySpark using AI-assisted tools (GitHub Copilot / Databricks Genie) with thorough manual review and validation.
Ensure migrated PySpark code matches original business logic, data quality standards, and pipeline performance.
Resolve Serverless incompatibilities — custom JARs, JDBC drivers, DBFS paths, streaming triggers, and unsupported Spark configurations.
Migrate Databricks job cluster definitions to Serverless compute with updated library and parameter configurations.
Perform data validation and runtime benchmarking — row counts, schema checks, and side-by-side output comparison.
Document migration changes, known issues, and post-migration runbook for production cutover.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: RTX1bc3ac
Position Id: 9062919
Posted 2 days ago

Company Info

About cloudingest inc

CloudIngest is a full-service tech software firm. We possess extensive practical experience in Management, Business, and Economics. We stay up-to-date on emerging trends in terms of both the evolving cloud-based tech stack and client considerations in terms of Financial Billing and Payments. Our team consists of Client Intake Managers, Project Managers, Software Developers, Quality Assurance, and Integrative Solution and NLP Specialists all of whom are fully-equipped and ready to interface with members of your existing team. We are experienced working with client-side Project Managers as well as Designers and C-Suite business executives.

Go to company profile

Contact the job poster

Sandy Smyth

Recruiter @ cloudingest inc

View Profile

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Dallas, Texas

•

2d ago

The Databricks Administrator is responsible for managing, securing, and optimizing Databricks workspaces and clusters across enterprise environments. Key Responsibilities: Provision, configure, and monitor Databricks clusters, jobs, and workflows. Set up and manage multi-tenant workspaces, cross-workspace access, and authentication. Implement and enforce Unity Catalog for data governance, lineage, and fine-grained access control. Manage user/group provisioning, role-based access, and workspace i

Easy Apply

Contract