Role : Data Engineering Lead
Location : Fremont, CA
Duration : 12 months
Job Description
We are looking for an experienced Data Engineering Lead with 10+ years of experience in designing, developing, and optimizing scalable data engineering solutions. The ideal candidate should have strong hands-on expertise in ETL/ELT development, SQL, PySpark, Databricks, Apache Spark, data modeling, and Medallion Architecture.
The candidate should be capable of independently driving data engineering initiatives, building robust data pipelines, and delivering high-quality solutions in a fast-paced environment.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines for structured and unstructured data.
- Build and optimize data processing solutions using Databricks, Apache Spark, PySpark, and Spark SQL.
- Develop efficient data integration solutions using APIs and various data sources.
- Design and implement database schemas, dimensional models, and data warehouse solutions.
- Implement and maintain Medallion Architecture (Bronze, Silver, and Gold layers).
- Optimize data pipelines, SQL queries, and Spark workloads for performance and scalability.
- Follow data warehousing best practices, development standards, and engineering methodologies.
- Independently manage technical deliverables and ensure timely project execution.
Required Technical Skills
- 10+ years of overall experience in Data Engineering.
- Strong hands-on experience with ETL/ELT development.
- Expertise in SQL, PySpark, and Spark SQL.
- Extensive experience with Databricks and Apache Spark.
- Strong understanding of structured and unstructured data processing.
- Experience with API integrations and data ingestion.
- Expertise in schema design and dimensional data modeling.
- Hands-on experience with Medallion Architecture.
- Strong knowledge of Data Warehouse concepts and best practices.
Preferred Candidate Profile
- Strong analytical and problem-solving skills.
- Self-motivated with an independent working approach.
- Ability to work in a fast-paced and dynamic environment.
- Strong ownership, communication, and delivery mindset.
