Description We are seeking a skilled and motivated Data Engineer to join our team, with deep hands-on experience building and optimizing data pipelines and lakehouse solutions in Databricks. In this role, you will collaborate with cross-functional teams to design, develop, and operate scalable, reliable data products that drive business value.
Key Responsibilities
- Design, build, and maintain batch and streaming data pipelines using Databricks (Spark, Delta Lake, Jobs/Workflows).
- Partner with data scientists, analysts, and application teams to deliver trusted, well-modeled data sets and features in the Databricks Lakehouse.
- Optimize Spark jobs (partitioning, caching, join strategies) and Databricks cluster configurations for performance, scalability, and cost.
- Implement data quality checks, observability, governance, and security controls (e.g., Unity Catalog, access policies) within Databricks.
- Troubleshoot and resolve pipeline failures, data issues, and production incidents; perform root-cause analysis and implement preventative improvements.
Requirements - Bachelor's degree in Computer Science, Engineering, or related field (or equivalent experience).
- Proven experience delivering production data engineering solutions in Databricks, including Apache Spark and Delta Lake on the Microsoft Azure cloud platform.
- Strong proficiency in SQL and Python (or Scala), including building ETL/ELT pipelines and writing efficient Spark code.
- Experience with data modeling and lakehouse/warehouse patterns (e.g., medallion architecture) and distributed data processing concepts.
- Excellent problem-solving skills and attention to detail.
- Strong communication and teamwork abilities.
Preferred Skills
- Hands-on experience with Databricks ecosystem capabilities such as Unity Catalog, Delta Live Tables, MLflow, and/or Databricks SQL.
- Experience implementing CI/CD and automated testing for Databricks notebooks/repos and deployment of Jobs/Workflows.
- Knowledge of data security, privacy, and compliance controls in cloud lakehouse environments (including role-based access and auditing).
Technology Doesn't Change the World, People Do.
Robert Half is the world's first and largest specialized talent solutions firm that connects highly qualified job seekers to opportunities at great companies. We offer contract, temporary and permanent placement solutions for finance and accounting, technology, marketing and creative, legal, and administrative and customer support roles.
Robert Half works to put you in the best position to succeed. We provide access to top jobs, competitive compensation and benefits, and free online training. Stay on top of every opportunity - whenever you choose - even on the go. Download the Robert Half app and get 1-tap apply, notifications of AI-matched jobs, and much more.
All applicants applying for U.S. job openings must be legally authorized to work in the United States. Benefits are available to contract/temporary professionals, including medical, vision, dental, and life and disability insurance. Hired contract/temporary professionals are also eligible to enroll in our company 401(k) plan. Visit roberthalf.gobenefits.net for more information.
2025 Robert Half. An Equal Opportunity Employer. M/F/Disability/Veterans. By clicking "Apply Now," you're agreeing to Robert Half's Terms of Use and Privacy Notice.
Never miss an opportunity! Create an alert based on the job you applied for.