Sr Databricks Engineer

Job Title: Sr Databricks Engineer

Location: San Jose, CA (5 days on-site)

 Hybrid role: 4 days onsite per week in San Jose, CA

Job Overview

We are seeking an experienced Senior Data Engineer to join our team in the San Jose Bay Area. The ideal candidate will have strong expertise in the Databricks Lakehouse platform, building and managing scalable data pipelines, working with notebooks, and implementing robust data monitoring solutions. This is a hybrid role requiring onsite presence for four days a week, along with close collaboration with offshore teams.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks (PySpark, Delta Lake, Workflows)
• Work extensively with Databricks notebooks for data engineering, transformation, and analysis
• Implement and manage data monitoring, logging, and alerting frameworks for data pipelines
• Write optimized SQL queries for large-scale data processing and analytics on Databricks
• Design and manage Databricks Workflows and/or Azure Data Factory (ADF)
• Ensure data quality, reliability, and performance across lakehouse layers (Bronze, Silver, Gold)
• Collaborate with cross-functional onsite and offshore teams to deliver end-to-end data solutions
• Troubleshoot and resolve complex data pipeline, performance, and scalability issues

Required Qualifications

• 8–10 years of experience in data engineering or related roles
• Strong hands-on experience with Databricks Lakehouse platform (PySpark, Delta Lake, Jobs/Workflows)
• Strong experience with Azure cloud services (ADLS, ADF, Key Vault, etc.)
• Proven expertise in building and managing scalable data pipelines and ETL/ELT frameworks
• Experience designing and managing using Databricks Workflows or ADF  
• Strong proficiency in SQL and data modeling (star schema, snowflake schema, dimensional modeling)
• Hands-on experience with notebook-based development environments (Databricks notebooks)
• Experience in data monitoring, logging, troubleshooting, and performance tuning
• Experience working in onsite/offshore delivery models
• Strong communication, analytical thinking, and problem-solving skills

Preferred Qualifications

• Strong experience in Data Modeling (dimensional modeling, star/snowflake schema design)
• Strong understanding of Data Warehousing concepts and architectures
• Hands-on experience with PySpark for large-scale distributed data processing
• Experience working with Azure Databricks in enterprise environments
• Experience with orchestration tools such as Apache Airflow, Azure Data Factory, or similar platforms
• Familiarity with big data technologies and distributed computing systems
• Prior experience in enterprise-scale data platform modernization projects

Work Arrangement

• Hybrid role: 4 days onsite per week in San Jose, CA
• Regular collaboration with offshore development and data engineering teams