DataBricks Engineer
Dice Job Match Score™
🔢 Crunching numbers...
Job Details
Skills
- Databricks
- Apache Kafka
- Python
- PySpark
- SQL
- Data Warehouse
- Data Modeling
- Data Governance
- DevOps
- Data Engineering
- Continuous Integration
- Continuous Delivery
Summary
Job Description: Databricks Engineer – Lead / Senior Lead
Job Title: Databricks Lead / Senior Lead Engineer
Experience: 8–12+ years (flexible based on level)
Job Type: Contract
Work Location: Lawrenseville, NJ
Job Summary
We are seeking an experienced Databricks Engineer for Lead and Senior Lead positions to design, develop, and optimize scalable data engineering solutions using Databricks, Apache Spark, Python, and SQL. The ideal candidate will have strong hands-on expertise in cloud-based data platforms, data pipeline development, data modeling, and performance optimization, along with the ability to lead technical teams and collaborate with business stakeholders.
Key Responsibilities
Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Python, and SQL.
Build and optimize ETL/ELT workflows for large-scale data processing.
Develop and manage Delta Lake tables, Lakehouse architectures, and data transformation frameworks.
Implement data ingestion, integration, cleansing, validation, and quality checks.
Optimize Spark jobs, cluster configurations, and query performance.
Work with cloud platforms such as AWS, Azure, or Google Cloud.
Implement workflow orchestration, monitoring, logging, and error-handling mechanisms.
Ensure data security, governance, access control, and compliance with organizational standards.
Collaborate with data architects, analysts, product teams, and business stakeholders.
Review code, establish engineering best practices, and mentor junior engineers.
Required Skills
Strong hands-on experience with Databricks and Apache Spark.
Advanced proficiency in PySpark, Python, and SQL.
Experience with Delta Lake, Lakehouse architecture, and ETL/ELT pipelines.
Strong understanding of data warehousing, data modeling, and distributed data processing.
Experience with cloud-based data platforms, preferably Azure Databricks or AWS Databricks.
Knowledge of CI/CD, Git, automated testing, and deployment practices.
Experience with data orchestration tools and workflow scheduling.
Strong analytical, problem-solving, and communication skills.
Role-Specific Expectations
1. Databricks Lead Engineer
Lead day-to-day technical development and delivery of data engineering solutions.
Guide a team of data engineers and conduct code reviews.
Translate technical requirements into scalable implementation plans.
Troubleshoot complex pipeline and performance issues.
Coordinate with cross-functional teams to ensure timely project delivery.
Maintain coding standards, documentation, and engineering best practices.
2. Databricks Senior Lead Engineer
Own the technical direction and architecture of complex Databricks solutions.
Lead multiple workstreams and guide teams through design and implementation.
Define engineering standards, reusable frameworks, and performance optimization strategies.
Drive cloud data platform modernization and large-scale data transformation initiatives.
Collaborate with architects and senior stakeholders on technical roadmaps and solution design.
Manage technical risks, delivery dependencies, and escalations.
Mentor technical leads and senior engineers and ensure high-quality delivery.
Preferred Qualifications
Databricks certifications or relevant cloud certifications.
Experience with Azure Data Factory, AWS Glue, or similar orchestration and integration tools.
Knowledge of streaming technologies such as Kafka or Structured Streaming.
Experience with Unity Catalog, data governance, and access management.
Familiarity with Terraform, DevOps, and infrastructure automation.
- Dice Id: 91173647
- Position Id: 9143506
- Posted 1 day ago
Company Info
About GState Consulting LLC
At GState Consulting, we are a strategic technology and business consulting firm committed to accelerating innovation and enabling long-term growth. We design and implement intelligent, purpose-driven solutions that transform how organizations operate and compete.
By combining industry insight with a results-oriented mindset, we help our clients lead confidently in an ever-evolving digital landscape.
Our Vision
To be the trusted partner that empowers organizations to innovate, adapt, and lead with confidence in a connected, technology-driven world.
