Job Description Databricks Tech Lead / Architect
Experience: 10 15 years
Location: Onsite (
New Jersey, USA)
Role Overview
We are looking for an experienced Databricks Tech Lead / Architect with 10 15 years of
experience in data engineering, data transformation, ETL/ELT, and modern cloud data
platforms. The candidate will be responsible for designing and leading scalable data
solutions using Databricks, Apache Spark, Delta Lake, SQL, Python and cloud data
services.
Key Responsibilities
Design and architect end-to-end data engineering and data transformation solutions
using Databricks.
Lead the development of scalable ETL/ELT pipelines for batch and near-real-time
data processing.
Develop robust data transformation frameworks using Databricks, Apache Spark,
PySpark and SQL.
Design and implement Delta Lake / Lakehouse architectures, including data
ingestion, transformation, optimization and consumption layers.
Define architecture patterns for bronze, silver and gold data layers and implement
appropriate data modelling strategies.
Design scalable ingestion frameworks for structured and unstructured data from
databases, APIs, files, cloud storage and enterprise applications.
Develop reusable frameworks for data cleansing, transformation, enrichment,
validation and reconciliation.
Optimize Spark workloads, Databricks clusters, SQL queries and data pipelines for
performance and cost efficiency.
Implement incremental processing, CDC, SCD Type 1/2, partitioning, caching and
data optimization strategies.
Define and implement data quality, data validation, exception handling and
monitoring mechanisms.
Establish engineering standards around coding, CI/CD, version control, testing,
deployment and operational support.
Work with cloud platforms such as AWS/Azure/Google Cloud Platform and integrate Databricks with
cloud-native data services.
Lead technical design discussions, architecture reviews and proof-of-concepts.
Mentor and guide data engineers and provide technical leadership across projects.
Collaborate with enterprise/data architects to ensure alignment with broader data
platform and integration architecture.
Required Technical Skills
Databricks & Spark
- Strong hands-on experience with Databricks and Apache Spark.
- Excellent knowledge of PySpark and Spark SQL.
- Strong understanding of Delta Lake and Lakehouse architecture.
- Experience with Databricks Workflows, Jobs, notebooks, clusters and
performance optimization.
Data Engineering / ETL / ELT
- Strong understanding of ETL and ELT architectures and design patterns.
- Experience designing complex data transformation pipelines.
- Strong SQL skills and experience with large-scale data processing.
- Experience with:
- Incremental data processing
- CDC
- SCD Type 1 / Type 2
- Data reconciliation
- Data quality and validation
- Error handling and restartability
- Metadata-driven pipelines
- Batch and near-real-time processing
Data Architecture
- Strong understanding of Data Lake, Data Warehouse and Lakehouse
architectures.
- Experience with dimensional modelling, star/snowflake schemas and data
marts.
- Understanding of data partitioning, file formats and storage optimization.
- Experience working with Parquet, JSON, CSV, Avro and other data formats.
Experience with one or more major cloud platforms:
- AWS: S3, Glue, Lambda, EMR, Redshift, IAM
- Azure: ADLS, Data Factory, Synapse, Event Hub
- Google Cloud Platform: GCS, BigQuery, Dataflow, Pub/Sub
Experience integrating Databricks with enterprise applications, databases, APIs and
cloud storage platforms.
DevOps / Engineering Practices
- Git-based development and branching strategies.
- CI/CD using tools such as Azure DevOps, GitHub Actions, Jenkins or GitLab.
- Experience with automated testing and deployment of Databricks
notebooks/jobs.
- Infrastructure-as-Code exposure using Terraform is desirable.
- Strong understanding of SDLC, Agile/Scrum and production support
processes.
.webp?format=webp)
Never miss an opportunity! Create an alert based on the job you applied for.
