Lead Databricks Engineer

Remote • Posted 1 hour ago • Updated 1 hour ago
Contract W2
Contract Independent
Occasional Travel Required
Remote
Depends on Experience
Company Branding Image
Fitment

Dice Job Match Score™

🤯 Applying directly to the forehead...

Job Details

Skills

  • Databricks
  • Delta Lake
  • Lakehouse
  • AWS
  • Power BI
  • Lead
  • python
  • pyspark
  • Amazon Redshift
  • Amazon Kinesis
  • ETL
  • DevOps
  • SQL

Summary

Title: Lead Databricks Engineer

Location: Remote

Hire Type: Contract To Hire

Once in 3/6 months for Pl Planning visit to DC metro area.

Job Description:

We are looking for a hands-on Databricks Engineer with strong AWS experience and good expertise in Data Analytics, including tools like Power BI and Tableau to design, build, and optimize scalable data pipelines and Lakehouse solutions. The role focuses on implementing robust batch and streaming data solutions using Databricks, Delta Lake, and AWS cloud-native services, ensuring high performance, scalability, and security.

Key Responsibilities

  • Build and maintain end-to-end data pipelines using Databricks, Delta Lake, and AWS services
  • Develop batch, real-time, and streaming data processing workflows
  • Implement data ingestion, transformation, curation, and storage pipelines
  • Build and optimize large-scale PySpark and SQL-based jobs in Databricks
  • Enable real-time data processing using Kafka, AWS Kinesis, or similar streaming tools

Data Lakehouse Implementation

  • Work on Databricks-based lakehouse architecture using Delta Lake
  • Implement scalable and optimized data storage and processing frameworks
  • Ensure data quality, consistency, and reliability across pipelines
  • Support metadata management, data lineage, and governance implementation.
  • Cloud & Platform Engineering (AWS)
  • Work with AWS services such as S3, Glue, Lambda, Kinesis, and Redshift
  • Ensure pipelines are scalable, secure, and cost-optimized in AWS environments
  • Implement security controls including RBAC, encryption, and data masking

Optimization & Best Practices

  • Tune Spark jobs for performance and cost efficiency
  • Monitor and troubleshoot data pipeline issues in production
  • Follow CI/CD and DevOps practices for deploying data engineering solutions
  • Ensure adherence to data engineering standards and best practices

Technical Leadership & Architecture

  • Lead the design and implementation of scalable, end‑to‑end data pipelines using Databricks, Delta Lake, and AWS services.
  • Architect and oversee batch, real‑time, and streaming data processing frameworks.
  • Drive the adoption of lakehouse best practices, ensuring robust data quality, governance, and lineage.
  • Provide technical direction on PySpark, SQL, and distributed data processing optimization.
  • Guide the team in implementing secure, cost‑efficient, and high‑performance data solutions on AWS.

Cross-Functional Collaboration

  • Partner with BI, analytics, and data science teams to deliver curated, high‑quality datasets for reporting and ML use cases.
  • Work closely with business stakeholders to translate requirements into scalable data engineering solutions.
  • Coordinate with cloud, security, and platform teams to ensure compliance, governance, and operational excellence.
  • Expertise in Data Analytics tools like Power BI, Tableau.

Required Skills & Qualifications:

  • Strong hands-on experience with Databricks.
  • Proficiency in Python, PySpark, and SQL
  • Strong experience in AWS cloud services (S3, Glue, Lambda, Kinesis, Redshift)
  • Experience building ETL/ELT data pipelines
  • Strong understanding of Delta Lake and lakehouse concepts
  • Experience with streaming and batch data processing
  • Knowledge of CI/CD tools and Git
  • Strong troubleshooting and performance tuning skills
  • Expertise in Data Analytics tools like Power BI, Tableau.
  • IaC (Terraform/CloudFormation)
  • Data quality & observability frameworks
  • Deeper Databricks-specific features (DLT, Unity Catalog, Workflows)
  • Security & compliance depth
  • DevOps tooling specifics
  • Leadership/communication expectations
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: 10114734
  • Position Id: 8969248
  • Posted 1 hour ago

Company Info

About Isoftech Inc

With over 15 years of experience in providing next-generation technology, we help companies align their ITgoals to their business objectivities. Headquartered in Chantilly, VA, ISoftech serves clients throughout the US. With a balanced view of strategy, people, process, and technology, we work with companies to understand their unique business needs, to develop and implement practical technology solutions. Clients benefit from ISoftech’s commitment to excellence, our responsiveness, flexibility and authentic desire to help them meet, and often exceed, their goals.

We as a firm will help clients optimize their goals, improve their operations, and accelerate their organizational performance to greater heights. Our turnkey solutions include Application Development and Modernization, Enterprise Architecture, Cloud Migration, Operation and Maintenance, Big Data and Analytics and Quality Engineering Services. ISoftech Inc’s portfolio of Clients include Commercial and Government Clients across United States.

ISoftech also provides product development and consulting services to help organizations actualize their goals. ISoftech has a well-informed view of the industry that enables us to properly identify a winning approach and process, no matter the industry. Our frameworks, tools and domain expertise are our key to building strong industry relations.

About_Company_OneAbout_Company_Two
Create job alert
Set job alertNever miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Remote

13d ago

Easy Apply

Contract

Depends on Experience

Remote

25d ago

Easy Apply

Contract

Depends on Experience

Search all similar jobs