Lead Data Engineer

Remote • Posted 30+ days ago • Updated 13 days ago
Contract W2
Contract Independent
No Travel Required
Remote
Depends on Experience
Fitment

Dice Job Match Score™

📊 Calculating match score...

Job Details

Skills

  • PySpark
  • Databricks
  • S3
  • ETL
  • SQL
  • Spark
  • CI/CD
  • Delta Lake
  • CloudWatch

Summary

Role : Lead Data Engineer
Location : Remote
Job Description
  • Design, develop, and maintain ETL/ELT pipelines using PySpark on Databricks.
  • Build and optimize batch and streaming data pipelines.
  • Implement Delta Lake solutions (Delta tables, time travel, ACID transactions).
  • Collaborate with data scientists, analysts, and architects to deliver analytics-ready datasets.
  • Optimize Spark jobs for performance, scalability, and cost.
  • Integrate data from multiple sources (RDBMS, APIs, files, cloud storage).
  • Implement data quality checks, validation, and monitoring.
  • Manage Databricks notebooks, jobs, clusters, and workflows.
  • Follow data governance, security, and compliance standards.
  • Participate in code reviews and contribute to best practices.
Qualifications
  • Hands-on experience with Data Frames, RDDs, joins, transformations, and actions within PySpark.
  • Proven experience leading teams and mentoring engineers.
  • Job optimization, cluster configuration, repartitioning, and Shuffle mechanics in Databricks.
  • S3 buckets, IAM, CloudWatch, and integration with Databricks and AWS.
  • Strong query skills for analytics and ETL with SQL.
  • Performance tuning: Partitioning, caching, broadcast joins, and skew handling.
  • Delta Lake, Medallion Architecture, Spark Streaming, Spark ML, and CI/CD pipelines.
  • ETL/ELT design patterns.
  • Handling large-scale structured and semi-structured data.
  • Performance tuning (partitioning, caching, broadcast joins).
  • Understanding of data warehousing concepts.
  • Excellent communication and stakeholder management skills.
  • Ability to work in Agile delivery environments.
  • Ownership mindset and delivery-focused approach.
  • Strong technical decision-making and problem-solving skills.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: 80122034
  • Position Id: 8863744
  • Posted 30+ days ago
Create job alert
Set job alertNever miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Remote or Almont, Colorado

Today

Contract

Remote

Today

Full-time

Remote or Bethlehem, Pennsylvania

Today

Full-time

USD 99,150.00 - 162,885.00 per year

Remote

26d ago

Easy Apply

Contract

Depends on Experience

Search all similar jobs