Lead Databricks Engineer

Lead Data Engineer

Location- San Jose, CA- Onsite  

Contract role  

 Requirements

 We are seeking a Lead Data Engineer with expertise in Databricks and Data Warehousing to drive data architecture, pipeline development, and optimization efforts. The ideal candidate will play a key role in designing scalable solutions, implementing best practices, and leading data initiatives within a dynamic and collaborative environment.

Key Responsibilities:

• Design, build, and optimize scalable data pipelines using Databricks, Apache Spark, and Azure technologies.
• Architect data warehousing solutions, ensuring seamless integration with cloud platforms and structured/unstructured data sources.
• Collaborate with business stakeholders to understand data needs and develop high-performance analytical solutions.
• Implement ETL/ELT processes leveraging cloud-based technologies such as Azure Data Factory, Snowflake, and Delta Lake.
• Ensure data quality, governance, and security compliance while managing large datasets efficiently.
• Drive performance tuning and optimization for data pipelines, ensuring efficiency across systems.
• Work closely with cross-functional teams to support machine learning and advanced analytics initiatives.
• Provide technical leadership and mentorship to junior data engineers, fostering a culture of innovation and continuous improvement.
• Stay updated on emerging data technologies and recommend strategies to enhance existing architectures.

Qualifications:

• 8+ years of experience in data engineering, big data processing, and cloud-based solutions.
• Strong expertise in Databricks, Spark (PySpark/SQL), and Delta Lake architecture.
• Proven experience in designing and managing data warehouses using Snowflake, Azure Synapse, or equivalent technologies.
• Deep understanding of data modeling, SQL, and performance optimization.
• Hands-on experience with Azure Data Factory, Event Hubs, and cloud-based ETL processes.
• Solid knowledge of real-time streaming technologies (Kafka, Azure Stream Analytics, or similar).
• Familiarity with ML/AI data pipelines and feature engineering best practices.
• Strong communication and collaboration skills, with experience working in fast-paced, enterprise environments.