Databricks Data Engineer

Bedminster, NJ, US • Posted 30+ days ago • Updated 6 hours ago

Full Time

On-site

Fitment

Dice Job Match Score™

📋 Comparing job requirements...

Job Details

Skills

Data Lake
Data Processing
Workflow
Scalability
Data Quality
Documentation
Data-flow Diagrams
Management
Computer Science
Data Science
Software Engineering
Information Systems
Databricks
Data Structure
Data Storage
Change Data Capture
Python
Star Schema
Dimensional Modeling
ELT
Extract
Transform
Load
Writing
SQL
PL/SQL
Relational Databases
Oracle
NoSQL
Database
MongoDB
Cosmos-Db
Cloud Computing
Continuous Integration
Continuous Delivery
Git
Microsoft Azure
DevOps
Storage
Apache Parquet
Apache Avro

Summary

Databricks Data Engineer

Responsibilities:

Assist with leading the team's transition to the Databricks platform and utilize the newer features of Delta Live Tables, Workflows etc.
Design and develop data pipelines that extract data from Oracle, load it into the data lake, transform it into the desired format, and load it into Databricks data lakehouse.
Optimize data pipelines and data processing workflows for performance, scalability, and efficiency.
Implement data quality checks and validations within data pipelines to ensure the accuracy, consistency, and completeness of data.
Help create and maintain documentation for data mappings, data definitions, architecture and data flow diagrams.
Build proof-of-concepts to determine viability of possible new processes and technologies.
Deploy and manage code in non-prod and prod environments.
Investigate and troubleshoot data related issues and fix or provide solutions to fix defects.
Identify and resolve performance bottlenecks, which could include suggesting ways to optimize and performance tune databases and queries to enhance query performance.

Requirements:

Bachelor's Degree in Computer Science, Data Science, Software Engineering, Information Systems, or related quantitative field.
4+ years of experience working as a Data Engineer, ETL Engineer, Data/ETL Architect or similar roles.
Current/active Databricks Data Engineer/Analyst certification a BIG plus.
3+ years working with Databricks with knowledge and expertise of data structures, data storage and change data capture gained from prior production implementations of data pipelines, optimizations, and best practices.
Solid continuous experience in Python.
3+ years of experience in Kimball dimensional modeling (star-schema comprising of facts, type1 and type2 dimensions, aggregates, etc.) with solid understanding of ELT/ETL.
3+ years of solid experience writing SQL and PL/SQL code.
2+ years of experience with Airflow.
3+ years of experience working with relational databases (Oracle preferred).
2+ years of experience working with NoSQL databases: MongoDB, Cosmos DB, DocumentDB or similar.
2+ years of cloud experience (Azure preferred).
Experience with CI/CD utilizing git/Azure DevOps.
Experience with storage formats including Parquet/Arrow/Avro.

#RecruitPS

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 90764204
Position Id: 648
Posted 30+ days ago

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

New York, New York

•

Today

Software Guidance & Assistance, Inc., (SGA), is searching for a Senior Data Platform Engineer - Databricks (SaaS on AWS) for a contract assignment with one of our premier Financial Services clients in New York City, NY. Responsibilities : Engineer and scale a Databricks (SaaS on AWS) platform supporting enterprise analytics and AI/ML workloads. Design and automate secure cloud infrastructure using Terraform (Infrastructure as Code) within AWS. Build and optimize production data workflows

Contract

Part-Time Senior Data Platform Engineer - Azure Databricks & API Integration

Lake Success, New York

•

Today

Job Description FlexStaff is seeking a Senior Data Platform Engineer - Azure Databricks & API Integration (PHI) This is a temporary role, for a one -year assignment Part-Time - approximately 20 hours/week Only candidates who reside in New York State can be considered. The Senior Data Platform Engineer will lead the operation, stability, and evolution of our client's Azure Databricks environment and the API integrations that support enterprise data exchange involving Protected Health Informatio

Part-time

Senior Data Engineer – Databricks / PySpark / Delta Lake

Remote

•

26d ago

Job Title: Senior Data Engineer Databricks, PySpark, Delta Lake Location: USA (100% Remote) Duration: 3 Months (Initial Contract, with potential extension) Experience Required: 10+ Years Role OverviewWe are seeking a highly experienced Senior Data Engineer who can take ownership of designing, building, and optimizing enterprise-scale data pipelines on Databricks using PySpark and Delta Lake within an Azure cloud environment. This role requires strong hands-on expertise and the ability to delive

Easy Apply

Full-time

50 - 55

Data Bricks Data Engineer

Remote

•

8d ago

Title: Data Bricks Data Engineer Location: - Myrtle Point, OR 97458 (100% Remote) Hire Type: W2 Contract Job Summary: Seeking an experienced Databricks Data Engineer with strong hands-on expertise in Databricks administration and engineering. The ideal candidate will be responsible for designing, developing, and managing scalable data solutions in a cloud-based environment (Azure or AWS). Required Skills: Strong hands-on experience as a Databricks Administrator and Data Engineer. Experience man