Lead Data Engineer

Remote • Posted 1 day ago • Updated 1 hour ago

Contract W2

Contract Independent

No Travel Required

Remote

Depends on Experience

Fitment

Dice Job Match Score™

🧠 Analyzing your skills...

Job Details

Skills

PySpark
Databricks
S3
ETL
SQL
Spark
CI/CD
Delta Lake
CloudWatch

Summary

Role : Lead Data Engineer

Location : Remote

Job Description

Design, develop, and maintain ETL/ELT pipelines using PySpark on Databricks.
Build and optimize batch and streaming data pipelines.
Implement Delta Lake solutions (Delta tables, time travel, ACID transactions).
Collaborate with data scientists, analysts, and architects to deliver analytics-ready datasets.
Optimize Spark jobs for performance, scalability, and cost.
Integrate data from multiple sources (RDBMS, APIs, files, cloud storage).
Implement data quality checks, validation, and monitoring.
Manage Databricks notebooks, jobs, clusters, and workflows.
Follow data governance, security, and compliance standards.
Participate in code reviews and contribute to best practices.

Qualifications

Hands-on experience with Data Frames, RDDs, joins, transformations, and actions within PySpark.
Proven experience leading teams and mentoring engineers.
Job optimization, cluster configuration, repartitioning, and Shuffle mechanics in Databricks.
S3 buckets, IAM, CloudWatch, and integration with Databricks and AWS.
Strong query skills for analytics and ETL with SQL.
Performance tuning: Partitioning, caching, broadcast joins, and skew handling.
Delta Lake, Medallion Architecture, Spark Streaming, Spark ML, and CI/CD pipelines.
ETL/ELT design patterns.
Handling large-scale structured and semi-structured data.
Performance tuning (partitioning, caching, broadcast joins).
Understanding of data warehousing concepts.
Excellent communication and stakeholder management skills.
Ability to work in Agile delivery environments.
Ownership mindset and delivery-focused approach.
Strong technical decision-making and problem-solving skills.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 80122034
Position Id: 8863744
Posted 1 day ago

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Lead Data Engineer

Dice Job Match Score™

Job Details

Skills

Summary

Similar Jobs