Search Jobs | Dice.com

Job Details

Skills

HP
Data Engineering
API
Data Processing
SAP BI
Performance Tuning
Query Optimization
Orchestration
RIS
Batch Processing
Programming Languages
Python
DS
DirectShow
Scala
Storage
VBC
Regulatory Compliance
Optimization
Cost Management
DevOps
Version Control
Artificial Intelligence
T1
Pure Data
Large Language Models (LLMs)
Extract
Transform
Load
ELT
PySpark
Apache Spark
SQL
Salesforce.com
Unity
Data Governance
Access Control
Amazon Redshift
Amazon Kinesis
Step-Functions
Amazon S3
Encryption
Collaboration
Business Intelligence
Analytics
Machine Learning (ML)
Continuous Integration
Continuous Delivery
Git
Workflow
Management
Machine Learning Operations (ML Ops)
Quality Management
Quantum Mechanics
Real-time
Streaming
Generative Artificial Intelligence (AI)
Vector Databases
Databricks
Amazon Web Services
Data Analysis
Terraform
Cloud Computing

HP
Data Engineering
API
Data Processing
SAP BI
Performance Tuning
Query Optimization
Orchestration
RIS
Batch Processing
Programming Languages
Python
DS
DirectShow
Scala
Storage
VBC
Regulatory Compliance
Optimization
Cost Management
DevOps
Version Control
Artificial Intelligence
T1
Pure Data
Large Language Models (LLMs)
Extract
Transform
Load
ELT
PySpark
Apache Spark
SQL
Salesforce.com
Unity
Data Governance
Access Control
Amazon Redshift
Amazon Kinesis
Step-Functions
Amazon S3
Encryption
Collaboration
Business Intelligence
Analytics
Machine Learning (ML)
Continuous Integration
Continuous Delivery
Git
Workflow
Management
Machine Learning Operations (ML Ops)
Quality Management
Quantum Mechanics
Real-time
Streaming
Generative Artificial Intelligence (AI)
Vector Databases
Databricks
Amazon Web Services
Data Analysis
Terraform
Cloud Computing

Summary

Job Title: Senior Databricks Engineer

Location: Malvern, PA

Job Type: Full-Time

Experience: 10-15 Years

Job Summary

We are seeking a highly experienced Senior Databricks Engineer with deep expertise in Apache Spark, Databricks Lakehouse Platform, Delta Lake, AWS, and modern data engineering. The ideal candidate will design and build scalable, secure, and high-performance data pipelines while implementing enterprise-grade data governance, CI/CD, and MLOps practices. Experience with Generative AI, LLMs, and Vector Search is highly desirable.

Required Skills

Databricks & Apache Spark

10-15 years of experience in Data Engineering
Expert in:
- Databricks Lakehouse Platform
- Apache Spark
- Spark SQL
- PySpark
- DataFrame API
- Distributed Data Processing
- Spark Performance Tuning
- Query Optimization

Delta Lake & Lakehouse Architecture

Delta Lake
Medallion Architecture (Bronze, Silver, Gold)
Delta Live Tables (DLT)
ACID Transactions
Z-Ordering
Data Optimization
Time Travel
Schema Evolution

Data Pipelines & Orchestration

ETL / ELT Development
Databricks Workflows
Databricks Jobs
Delta Live Tables (DLT)
Workflow Automation
Batch Processing
Streaming Data Pipelines
Error Handling
Retry Mechanisms

Programming Languages

Python
PySpark
SQL
Scala (Preferred)

AWS Cloud & Storage

Amazon S3
Amazon Redshift
AWS Glue
Amazon Kinesis
AWS Step Functions
AWS IAM
AWS KMS
Cross-Account IAM Roles
S3 Bucket Policies

Databricks Governance & Security

Unity Catalog
Data Lineage
Row-Level Security
Column-Level Security
Data Governance
Secure Data Sharing
Compliance

Compute & Cost Optimization

Cluster Policies
Instance Profiles
Spot Instances
Compute Optimization
Cost Management

CI/CD & DevOps

Git
Databricks Git Folders
CI/CD Pipelines
Version Control
Deployment Automation

AI & MLOps

MLflow
Model Registry
Experiment Tracking
Feature Engineering
Large Language Models (LLMs)
Vector Search
Generative AI Integrations

Key Responsibilities

Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, Apache Spark, and Delta Lake
Build and optimize batch and streaming data pipelines processing data from multiple enterprise sources
Develop high-performance PySpark and Spark SQL solutions, optimizing distributed processing and execution plans
Implement Medallion Architecture (Bronze, Silver, Gold) using Delta Lake best practices
Design and automate resilient workflows using Databricks Workflows, Jobs, and Delta Live Tables (DLT)
Configure Unity Catalog for enterprise data governance, lineage, and fine-grained access controls
Integrate Databricks with AWS services including Amazon S3, Redshift, Glue, Kinesis, and Step Functions
Implement secure cloud architectures using IAM roles, S3 bucket policies, and AWS KMS customer-managed encryption keys
Optimize Databricks compute resources through cluster policies, instance profiles, and Spot Instances
Collaborate with Data Scientists, ML Engineers, Analysts, and BI teams to support analytics, machine learning, and Generative AI initiatives
Implement CI/CD pipelines, Git-based development workflows, and deployment automation
Utilize MLflow for experiment tracking, model management, and MLOps lifecycle support

Preferred Qualifications

Experience with real-time streaming architectures and event-driven data platforms
Hands-on experience with Generative AI, LLMs, Vector Databases, and Retrieval-Augmented Generation (RAG)
Databricks Certified Data Engineer Professional or Associate
AWS Certified Data Analytics or Solutions Architect certification
Experience with Terraform or Infrastructure-as-Code (IaC)
Experience supporting enterprise-scale cloud data platforms

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 91018020
Position Id: PDT - 11633-12768-1784638349
Posted 9 hours ago

Company Info

About Purple Drive Technologies LLC

Founded in 2007, Purple Drive started as a tech solutions firm and has grown into a full-service consulting and talent partner. We help businesses navigate complex technology challenges while connecting top professionals with career-defining opportunities.

We believe in transforming businesses through smart IT solutions and empowering technologists to grow their expertise through challenging projects and meaningful partnerships. Built on over 20 years of trusted relationships, we create success stories for both our clients and the talented professionals who drive innovation forward.

Fitment

Dice Job Match Score™

🔗 Matching skills to job...

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

AI Engineer / Lead AI Engineer

Santa Clara, California

•

Today

Job Description Must Have Technical/Functional Skills: 5+ years of experience in AI/ML. 1+ years of experience building Agentic AI solutions. Strong expertise in Python and modern software engineering practices. Hands-on experience with LangGraph, LangChain, LLMs, and prompt engineering. Experience architecting and deploying enterprise AI applications. Familiarity with Azure AI Foundry, AWS Bedrock, Google Gemini Enterprise, or similar platforms. Strong understanding of APIs, event-driven archi

Easy Apply

Full-time, Part-time, Contract, Third Party

Filter Results

Distance

Job Title: Senior Databricks Engineer

Job Summary

Required Skills

Databricks & Apache Spark

Delta Lake & Lakehouse Architecture

Data Pipelines & Orchestration

Programming Languages

AWS Cloud & Storage

Databricks Governance & Security

Compute & Cost Optimization

CI/CD & DevOps

AI & MLOps

Key Responsibilities

Preferred Qualifications

Senior Databricks Engineer

Job Details

Skills

Summary

Job Title: Senior Databricks Engineer

Job Summary

Required Skills

Databricks & Apache Spark

Delta Lake & Lakehouse Architecture

Data Pipelines & Orchestration

Programming Languages

AWS Cloud & Storage

Databricks Governance & Security

Compute & Cost Optimization

CI/CD & DevOps

AI & MLOps

Key Responsibilities

Preferred Qualifications

Company Info

About Purple Drive Technologies LLC

Dice Job Match Score™

Similar Jobs