Data Engineer

Description

We are looking for an experienced Data Engineer to join a team delivering modern data solutions in Poughkeepsie, New York. This Long-term Contract position focuses on building reliable, scalable data platforms in Databricks while supporting analytics needs across the business. The role calls for a hands-on engineer who can improve data performance, uphold governance standards, and partner effectively with cross-functional stakeholders in an agile environment.

Responsibilities:
Create and support scalable data pipelines in Databricks using Spark technologies such as PySpark or Scala to process and deliver high-quality data.
Develop lakehouse architectures on Azure Data Lake Storage Gen2 and ensure strong integration with Databricks for efficient data management.
Establish and monitor data quality controls and governance practices within the platform using validation methods and Delta Lake capabilities.
Investigate pipeline and application inefficiencies, then implement tuning strategies to improve Spark and Databricks performance.
Work closely with analysts and other stakeholders to translate business data needs into refined, analytics-ready datasets.
Automate ingestion, transformation, testing, and release processes, including integration with CI/CD workflows where appropriate.
Provide guidance to less experienced engineers by sharing best practices for Databricks development, optimization, and support.
Maintain clear technical documentation for notebooks, workflows, data models, configurations, and operational procedures.
Protect data assets by applying security controls and compliance standards across the Databricks environment.
Contribute to design sessions, solve complex data issues, and uphold change management and data integrity standards while delivering large assignments on schedule.

Requirements

Hands-on experience building data engineering solutions with Databricks and Apache Spark.
Strong programming ability in Python, including development of ETL and data transformation workflows.
Knowledge of lakehouse and big data technologies such as Delta Lake, Apache Hadoop, and Apache Kafka.
Experience working with Azure Data Lake Storage Gen2 or comparable cloud-based data storage platforms.
Ability to optimize distributed data processing jobs and troubleshoot performance issues in Spark environments.
Familiarity with data governance, data quality, and security practices for enterprise data platforms.
Comfortable working independently and collaborating with cross-functional teams in an agile delivery model.
Proven ability to analyze technical problems, break them into manageable components, and implement effective solutions.

Technology Doesn't Change the World, People Do.

Robert Half is the world's first and largest specialized talent solutions firm that connects highly qualified job seekers to opportunities at great companies. We offer contract, temporary and permanent placement solutions for finance and accounting, technology, marketing and creative, legal, and administrative and customer support roles.

Robert Half works to put you in the best position to succeed. We provide access to top jobs, competitive compensation and benefits, and free online training. Stay on top of every opportunity - whenever you choose - even on the go. Download the Robert Half app and get 1-tap apply, notifications of AI-matched jobs, and much more.

All applicants applying for U.S. job openings must be legally authorized to work in the United States. Benefits are available to contract/temporary professionals, including medical, vision, dental, and life and disability insurance. Hired contract/temporary professionals are also eligible to enroll in our company 401(k) plan. Visit roberthalf.gobenefits.net for more information.

2025 Robert Half. An Equal Opportunity Employer. M/F/Disability/Veterans. By clicking "Apply Now," you're agreeing to Robert Half's Terms of Use and Privacy Notice.