Data Engineer

Description

Position Overview

We are seeking a Data Engineer Engineer to support and enhance a Databricks-based data platform during its development phase. This role is focused on building reliable, scalable data solutions early in the lifecycle-not production firefighting.

The ideal candidate brings hands-on experience with Databricks, PySpark, Python, and a working understanding of Azure cloud services. You will partner closely with Data Engineering teams to ensure pipelines, notebooks, and workflows are designed for long-term scalability and production readiness.

Key Responsibilities

• Develop and enhance Databricks notebooks, jobs, and workflows
• Write and optimize PySpark and Python code for distributed data processing
• Assist in designing scalable and reliable data pipelines
• Apply Spark performance best practices: partitioning, caching, joins, file sizing
• Work with Delta Lake tables, schemas, and data models
• Perform data validation and quality checks during development cycles
• Support cluster configuration, sizing, and tuning for development workloads
• Identify performance bottlenecks early and recommend improvements
• Partner with Data Engineers to prepare solutions for future production rollout
• Document development standards, patterns, and best practices

Requirements

Required Skills & Qualifications

3+ years of hands-on experience working with:

Databricks & Spark

• Strong hands-on Databricks experience
• Proficiency with PySpark and Spark fundamentals
• Experience with Delta Lake
• Understanding of Spark execution (jobs, stages, tasks, shuffles)

Programming

• Strong Python development skills
• Ability to write clean, modular, reusable code

Cloud Experience (Azure Preferred)

• Experience working in Microsoft Azure
• Familiarity with:
• Azure Databricks
• Azure Data Lake Storage (ADLS)
• Azure Blob Storage
• Basic understanding of cloud resource usage and cost awareness

Development Practices

• Experience using Git and version control workflows
• Familiarity with Databricks Repos or similar tools
• Ability to perform testing and validation of data pipelines

Technology Doesn't Change the World, People Do.

Robert Half is the world's first and largest specialized talent solutions firm that connects highly qualified job seekers to opportunities at great companies. We offer contract, temporary and permanent placement solutions for finance and accounting, technology, marketing and creative, legal, and administrative and customer support roles.

Robert Half works to put you in the best position to succeed. We provide access to top jobs, competitive compensation and benefits, and free online training. Stay on top of every opportunity - whenever you choose - even on the go. Download the Robert Half app and get 1-tap apply, notifications of AI-matched jobs, and much more.

All applicants applying for U.S. job openings must be legally authorized to work in the United States. Benefits are available to contract/temporary professionals, including medical, vision, dental, and life and disability insurance. Hired contract/temporary professionals are also eligible to enroll in our company 401(k) plan. Visit roberthalf.gobenefits.net for more information.

2025 Robert Half. An Equal Opportunity Employer. M/F/Disability/Veterans. By clicking "Apply Now," you're agreeing to Robert Half's Terms of Use and Privacy Notice.