Sr. Data Fabric Engineer 5 days on-site

Senior Data Fabric Engineer 

Location: San Jose, CA | 5 days on-site

Contract role 

Job Description:

Role Overview

We are seeking a highly skilled Senior Data Engineer with a deep specialization in Apache Spark and the Microsoft Fabric ecosystem. The ideal candidate isn''t just a notebook user; you are an expert in developing, packaging, and deploying Spark-based JAR files (Scala/Java) to handle complex, high-scale data processing requirements.

You will be responsible for architecting robust ETL/ELT pipelines, optimizing Spark performance, and leveraging the full suite of Microsoft Fabric tools (OneLake, Lakehouse, and Data Factory) to drive our data strategy forward.

Key Responsibilities

• Custom Spark Development: Design and develop high-performance data processing applications using Scala or Java, compiled into JARs for execution on Spark clusters.

• Fabric Implementation: Architect and maintain end-to-end data solutions within Microsoft Fabric, utilizing Synapse Data Engineering and OneLake.

• Performance Tuning: Optimize Spark jobs by managing partitions, caching strategies, and memory management to ensure efficient resource utilization.

• Pipeline Orchestration: Build and automate sophisticated data workflows using Fabric Data Factory and Airflow (or similar).

• DevOps & CI/CD: Manage the lifecycle of Spark JARs through automated CI/CD pipelines (Azure DevOps/GitHub Actions), ensuring seamless deployment to Fabric environments.

• Data Modeling: Implement Medallion Architecture (Bronze/Silver/Gold) and maintain Delta Lake tables for ACID compliance and time travel capabilities.

Technical Requirements

Core Essentials

• Spark Expertise: 5+ years of experience with Apache Spark, specifically in writing and deploying compiled JAR files rather than solely relying on PySpark notebooks.

• Languages: Proficiency in Scala or Java (required) and Python/SQL (preferred).

• Platform: Hands-on experience with Microsoft Fabric or migrations from Azure Databricks/Synapse to Fabric.

• Storage: Deep understanding of Delta Lake format and Parquet optimization.