Senior Databricks Engineer

Description

We are looking for an experienced Senior Databricks Engineer to support enterprise data initiatives. This Long-term Contract position will focus on building and enhancing scalable data solutions, including extracting data from ServiceNow and structuring it for downstream ARDoc consumption within a broader enterprise architecture program. The role will partner with engineering, analytics, and platform teams to deliver reliable, governed, and high-performing data pipelines on Databricks.

Responsibilities:

Develop and maintain robust data pipelines in Databricks using Apache Spark to support enterprise reporting and analytical use cases.

Extract and transform data from ServiceNow, then create and manage structured tables that supply ARDoc as part of a larger architecture initiative.

Design layered data models using Delta Lake practices to improve scalability, data usability, and long-term maintainability.

Configure and manage Databricks jobs, workflows, and compute resources to improve processing efficiency and platform stability.

Investigate performance issues across Spark workloads and implement tuning strategies that improve speed, reliability, and cost control.

Establish data governance practices through cataloging, lineage visibility, and secure access controls within the Databricks environment.

Work closely with data scientists, analysts, and platform teams to deploy production-ready data assets and support advanced analytics needs.

Provide technical guidance to other engineers by promoting engineering standards, reviewing designs, and sharing platform best practices.

Requirements

At least 5 years of experience in data engineering working with large-scale or distributed data processing platforms.
Minimum 3 years of hands-on experience building solutions in Databricks within production environments.
Strong coding ability in Python, especially PySpark, with additional SQL proficiency for data transformation and analysis.
Solid knowledge of Apache Spark concepts, including execution behavior, performance tuning, and resource management.
Practical experience with Delta Lake features such as schema evolution, versioned data management, and reliable transactional processing.
Familiarity with cloud-based data ecosystems and orchestration tools used to schedule, monitor, and manage data workloads.
Experience using version control and CI/CD practices to support repeatable and controlled data engineering deployments.
ServiceNow knowledge is highly valued, especially for roles involving data extraction and integration into downstream platforms.

Technology Doesn't Change the World, People Do.

Robert Half is the world's first and largest specialized talent solutions firm that connects highly qualified job seekers to opportunities at great companies. We offer contract, temporary and permanent placement solutions for finance and accounting, technology, marketing and creative, legal, and administrative and customer support roles.

Robert Half works to put you in the best position to succeed. We provide access to top jobs, competitive compensation and benefits, and free online training. Stay on top of every opportunity - whenever you choose - even on the go. Download the Robert Half app and get 1-tap apply, notifications of AI-matched jobs, and much more.

All applicants applying for U.S. job openings must be legally authorized to work in the United States. Benefits are available to contract/temporary professionals, including medical, vision, dental, and life and disability insurance. Hired contract/temporary professionals are also eligible to enroll in our company 401(k) plan. Visit roberthalf.gobenefits.net for more information.

2025 Robert Half. An Equal Opportunity Employer. M/F/Disability/Veterans. By clicking "Apply Now," you're agreeing to Robert Half's Terms of Use and Privacy Notice.