Databricks Data Engineer

job summary:

We are seeking a experienced Databricks Data Engineer with expertise in data engineering, big data, and cloud solutions. The ideal candidate will have hands-on experience with on-prem to AWS private cloud migrations, Databricks PVC-to-SaaS migration, and modern data stack components such as S3, Spark, Airflow, Lakehouse architectures, and real-time databases. Must be willing to be on call periodically from 8am-2pm ET or 2pm-8pm ET Monday through Friday. Remote position.

location: Telecommute

job type: Solutions

salary: $90 - 110 per hour

work hours: 8am to 5pm

education: Bachelors



responsibilities:

In this role, you will design and build scalable, secure, and reusable data products and pipelines to collect, process, manage, analyze, and visualize large datasets across multiple platforms-transforming data into actionable insights-while also conducting testing across development, staging, and pre-production environments to ensure the reliability and quality of data workflows.

qualifications:

Proficiency in Collibra, Starburst, relational and NoSQL data stores, and expertise in data modeling techniques like star and dimensional modeling is essential. Specific skill mixes we need from the tech stack:

- Databricks AND (R or sparklyr) OR (Pyspark or Python)

- Databricks AND (Starburst OR Tableau)

- Databricks AND (Collibra OR Altreyx)

HANDS-ON Experience for the following:

- Databricks -Pyspark, Data Quality Framework, Building custom dashboards

- R/SparklyR - as a data analyst

Required Skills - Databricks, On-prem to AWS cloud migrations, Databricks PVC to SaaS migration, Big data, modern data stack components such as S3, Spark, Airflow, Lakehouse architectures, Collibra, Starburst, data modeling, Data ingestion, ci/cd, Testing Data & Workflows.

Job Duties - - Strong expertise in data platform ecosystems with proven experience in Databricks engineering and pipeline architecture, as well as hands-on knowledge of Collibra and Starburst.

- Databricks PVC to SaaS migration experienced, ensuring minimal disruption and high performance.

- Strong background in cloud migration projects, On-premise data systems to a private cloud architecture on AWS, ensuring minimal downtime and optimal performance.

- Provide strategic input on scaling Databricks infrastructure for performance and cost optimization.

- Design, develop and maintain robust, and efficient data pipelines to ingest, transform, catalog, and deliver curated, trusted and quality data from disparate data sources into Common Data Platform and making it available for comprehensive business analysis.

- Deliver high quality data products and services following Safe Agile Practices

- Good understanding of data architecture, information security, data governance, Develops processes.

- Proactively identifying and resolving issues with data pipelines and analytical data stores

- Deploying monitoring and alerting for data pipelines, data stores and implementing auto remediation wherever possible to ensure the availability and reliability of data systems.

- Employing security, testing and automation first strategy and adhering to data engineering best practices

- Collaborating with cross-functional teams, including product mgmt., data scientists, analysts, and business stakeholders, to understand their data requirements and provide them with the necessary infrastructure and tools.

- Keeping up with the latest trends and technologies which includes evaluating and recommending new tools, frameworks, and technologies to improve data engineering processes and efficiencies.

- Overall data engineering experience across traditional ETL & Big Data, either on-prem or Cloud

- Data engineering experience in AWS (any CFS2/EDS) highlighting the services/tools used.

- Experience building end to end data pipelines to ingest and process unstructured semistructured data using Spark architecture.

Job Requirements - - Bachelor's degree in computer science, Information Systems, or another related field or equivalent experience.

- Databricks, Data Engineering, AWS, On-prem to AWS cloud migrations, Databricks PVC to SaaS migration, Big data, modern data stack components such as S3, Spark, Airflow, Lakehouse architectures, Collibra, Starburst, relational and NoSQL, data modeling, Data ingestion, ci/cd, Testing Data & Workflows in test environments.

Equal Opportunity Employer: Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other legally protected group status.

At Randstad Digital, we welcome people of all abilities and want to ensure that our hiring and interview process meets the needs of all applicants. If you require a reasonable accommodation to make your application or interview experience a great one, please contact

Pay offered to a successful candidate will be based on several factors including the candidate's education, work experience, work location, specific job duties, certifications, etc. In addition, Randstad Digital offers a comprehensive benefits package, including: medical, prescription, dental, vision, AD&D, and life insurance offerings, short-term disability, and a 401K plan (all benefits are based on eligibility).

This posting is open for thirty (30) days.


It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability.