Azure Data Engineer (Must be Local to CA)

Azure Data Engineer

Milpitas, CA (Must be local to CA)

Contract

Key Responsibilities

Technical Leadership & Architecture

• Lead the design and implementation of Databricks Lakehouse architectures
• Define medallion architecture (Bronze, Silver, Gold layers) using Delta Lake
• Drive architectural decisions for batch and streaming data pipelines
• Establish coding standards, best practices, and reusable frameworks

Data Engineering & Databricks

• Design and build scalable ETL/ELT pipelines using Databricks (PySpark/SQL/Scala)
• Optimize Spark jobs for performance, reliability, and cost
• Implement Delta Lake features (ACID, time travel, schema enforcement)
• Develop and manage Databricks workflows, jobs, and clusters

Cloud & Platform Integration

• Architect Databricks solutions on Azure (preferred) or AWS
• Integrate Databricks with cloud storage and data services
• Azure: ADLS, ADF, Synapse
• AWS: S3, Glue, Redshift
• Enable BI and analytics consumption (Power BI, Tableau)

Governance, Security & DevOps

• Implement data governance using Unity Catalog
• Define RBAC, data access controls, and security best practices
• Enable CI/CD for Databricks using GitHub / Azure DevOps
• Use Infrastructure-as-Code (Terraform) for environment management

Leadership & Collaboration

• Lead, mentor, and grow data engineering teams
• Conduct design and code reviews
• Collaborate with Data Architects, Product Owners, and stakeholders
• Support production releases, monitoring, and incident resolution