Apply Now

Databricks Data Engineer

Bedminster, NJ, US • Posted 60+ days ago • Updated 14 hours ago

Full Time

On-site

Fitment

Dice Job Match Score™

⏳ Almost there, hang tight...

Job Details

Skills

Data Lake
Data Processing
Workflow
Scalability
Data Quality
Documentation
Data-flow Diagrams
Management
Computer Science
Data Science
Software Engineering
Information Systems
Databricks
Data Structure
Data Storage
Change Data Capture
Python
Star Schema
Dimensional Modeling
ELT
Extract
Transform
Load
Writing
SQL
PL/SQL
Relational Databases
Oracle
NoSQL
Database
MongoDB
Cosmos-Db
Cloud Computing
Continuous Integration
Continuous Delivery
Git
Microsoft Azure
DevOps
Storage
Apache Parquet
Apache Avro

Summary

Databricks Data Engineer

Responsibilities:

Assist with leading the team's transition to the Databricks platform and utilize the newer features of Delta Live Tables, Workflows etc.
Design and develop data pipelines that extract data from Oracle, load it into the data lake, transform it into the desired format, and load it into Databricks data lakehouse.
Optimize data pipelines and data processing workflows for performance, scalability, and efficiency.
Implement data quality checks and validations within data pipelines to ensure the accuracy, consistency, and completeness of data.
Help create and maintain documentation for data mappings, data definitions, architecture and data flow diagrams.
Build proof-of-concepts to determine viability of possible new processes and technologies.
Deploy and manage code in non-prod and prod environments.
Investigate and troubleshoot data related issues and fix or provide solutions to fix defects.
Identify and resolve performance bottlenecks, which could include suggesting ways to optimize and performance tune databases and queries to enhance query performance.

Requirements:

Bachelor's Degree in Computer Science, Data Science, Software Engineering, Information Systems, or related quantitative field.
4+ years of experience working as a Data Engineer, ETL Engineer, Data/ETL Architect or similar roles.
Current/active Databricks Data Engineer/Analyst certification a BIG plus.
3+ years working with Databricks with knowledge and expertise of data structures, data storage and change data capture gained from prior production implementations of data pipelines, optimizations, and best practices.
Solid continuous experience in Python.
3+ years of experience in Kimball dimensional modeling (star-schema comprising of facts, type1 and type2 dimensions, aggregates, etc.) with solid understanding of ELT/ETL.
3+ years of solid experience writing SQL and PL/SQL code.
2+ years of experience with Airflow.
3+ years of experience working with relational databases (Oracle preferred).
2+ years of experience working with NoSQL databases: MongoDB, Cosmos DB, DocumentDB or similar.
2+ years of cloud experience (Azure preferred).
Experience with CI/CD utilizing git/Azure DevOps.
Experience with storage formats including Parquet/Arrow/Avro.

#RecruitPS

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 90764204
Position Id: 648
Posted 30+ days ago

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Lead, Data Platform Engineer

Raritan, New Jersey

•

Today

At Johnson & Johnson, we believe health is everything. Our strength in healthcare innovation empowers us to build a world where complex diseases are prevented, treated, and cured, where treatments are smarter and less invasive, and solutions are personal. Through our expertise in Innovative Medicine and MedTech, we are uniquely positioned to innovate across the full spectrum of healthcare solutions today to deliver the breakthroughs of tomorrow, and profoundly impact health for humanity. Learn m

Full-time

USD 132,000.00 - 150,000.00 per year

Google Cloud Platform Data Engineer - Onsite Role

South Plainfield, New Jersey

•

7d ago

Role Overview Senior Data Engineer with 10+ years of experience in Google Cloud Platform and pharma domain. Responsible for building scalable data pipelines and supporting analytics, reporting, and ML use cases. Key ResponsibilitiesBuild and maintain data pipelines using Google Cloud Platform (BigQuery, Dataflow, Pub/Sub, Cloud Storage).Develop and optimize ETL/ELT workflows.Collaborate with analytics and business teams to deliver reliable data.Design data models and ensure data quality and per

Easy Apply

Full-time, Third Party

Depends on Experience

Senior Data Engineer Azure Orchestration & Snowflake Data Platform (GxP)

New Jersey

•

13d ago

Position: Senior Data Engineer Azure Orchestration & Snowflake Data Platform (GxP) Location: Somerset, NJ (Onsite) Duration: 6+ months (CTH) Position Summary We are seeking a Senior Data Engineer to build and operate enterprise-grade data pipelines using Microsoft Azure as the orchestration layer and Snowflake as the enterprise data lake/warehouse.This role focuses on reliable ETL/ELT, push/pull data integration patterns, and scalable platform engineering in a GxP / validated environment. Key R

Easy Apply

Contract

$50 - $60

Senior Data Engineer- NJ

Woodbridge Township, New Jersey

•

Today

Job#: 3030673 Job Description: Senior Data Engineer Location: Iselin, NJ (Hybrid) Employment Type: 12 Months Contract Pay Rate: $58/hour - $61/hour Role Overview In this contingent resource assignment, the Software Engineer will consult on and participate in moderately complex initiatives within Software Engineering. The role involves contributing to large-scale planning for data and platform engineering, analyzing complex challenges, and resolving issues by applying a solid understanding

Easy Apply

Full-time

USD 58.00 - 61.00 per hour

Search all similar jobs