Senior Databricks Engineer

Job Title: Senior Databricks Engineer

Location: Malvern, PA

Job Type: Full-Time

Experience: 10-15 Years

Job Summary

We are seeking a highly experienced Senior Databricks Engineer with deep expertise in Apache Spark, Databricks Lakehouse Platform, Delta Lake, AWS, and modern data engineering. The ideal candidate will design and build scalable, secure, and high-performance data pipelines while implementing enterprise-grade data governance, CI/CD, and MLOps practices. Experience with Generative AI, LLMs, and Vector Search is highly desirable.

Required Skills

Databricks & Apache Spark

• 10-15 years of experience in Data Engineering
• Expert in:
  • Databricks Lakehouse Platform
  • Apache Spark
  • Spark SQL
  • PySpark
  • DataFrame API
  • Distributed Data Processing
  • Spark Performance Tuning
  • Query Optimization

Delta Lake & Lakehouse Architecture

• Delta Lake
• Medallion Architecture (Bronze, Silver, Gold)
• Delta Live Tables (DLT)
• ACID Transactions
• Z-Ordering
• Data Optimization
• Time Travel
• Schema Evolution

Data Pipelines & Orchestration

• ETL / ELT Development
• Databricks Workflows
• Databricks Jobs
• Delta Live Tables (DLT)
• Workflow Automation
• Batch Processing
• Streaming Data Pipelines
• Error Handling
• Retry Mechanisms

Programming Languages

• Python
• PySpark
• SQL
• Scala (Preferred)

AWS Cloud & Storage

• Amazon S3
• Amazon Redshift
• AWS Glue
• Amazon Kinesis
• AWS Step Functions
• AWS IAM
• AWS KMS
• Cross-Account IAM Roles
• S3 Bucket Policies

Databricks Governance & Security

• Unity Catalog
• Data Lineage
• Row-Level Security
• Column-Level Security
• Data Governance
• Secure Data Sharing
• Compliance

Compute & Cost Optimization

• Cluster Policies
• Instance Profiles
• Spot Instances
• Compute Optimization
• Cost Management

CI/CD & DevOps

• Git
• Databricks Git Folders
• CI/CD Pipelines
• Version Control
• Deployment Automation

AI & MLOps

• MLflow
• Model Registry
• Experiment Tracking
• Feature Engineering
• Large Language Models (LLMs)
• Vector Search
• Generative AI Integrations

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, Apache Spark, and Delta Lake
• Build and optimize batch and streaming data pipelines processing data from multiple enterprise sources
• Develop high-performance PySpark and Spark SQL solutions, optimizing distributed processing and execution plans
• Implement Medallion Architecture (Bronze, Silver, Gold) using Delta Lake best practices
• Design and automate resilient workflows using Databricks Workflows, Jobs, and Delta Live Tables (DLT)
• Configure Unity Catalog for enterprise data governance, lineage, and fine-grained access controls
• Integrate Databricks with AWS services including Amazon S3, Redshift, Glue, Kinesis, and Step Functions
• Implement secure cloud architectures using IAM roles, S3 bucket policies, and AWS KMS customer-managed encryption keys
• Optimize Databricks compute resources through cluster policies, instance profiles, and Spot Instances
• Collaborate with Data Scientists, ML Engineers, Analysts, and BI teams to support analytics, machine learning, and Generative AI initiatives
• Implement CI/CD pipelines, Git-based development workflows, and deployment automation
• Utilize MLflow for experiment tracking, model management, and MLOps lifecycle support

Preferred Qualifications

• Experience with real-time streaming architectures and event-driven data platforms
• Hands-on experience with Generative AI, LLMs, Vector Databases, and Retrieval-Augmented Generation (RAG)
• Databricks Certified Data Engineer Professional or Associate
• AWS Certified Data Analytics or Solutions Architect certification
• Experience with Terraform or Infrastructure-as-Code (IaC)
• Experience supporting enterprise-scale cloud data platforms