Senior Databricks Engineer / Architect (Remote)

Job Title: Senior Databricks Engineer / Architect
Location: 100% Remote
Job Type: Long-term Contract

Job Description.

Position Overview
We are seeking a Senior Databricks Engineer to join our enterprise data platform team supporting one of our clients. In this role, you will be responsible for designing, building, and optimizing modern Data Lakehouse solutions using the Databricks platform and automating infrastructure deployment via Terraform. The ideal candidate possesses deep expertise in high-throughput data pipelines, cloud architecture (Azure/AWS), infrastructure-as-code, and advanced data lake governance.

Key Responsibilities
Databricks Architecture & Engineering: Architect, build, and maintain production-grade data pipelines using Spark, Delta Lake, Databricks Workflows, and Unity Catalog.
Infrastructure as Code (IaC): Automate and manage Databricks workspace configurations, clusters, security, and cloud resource provisioning using Terraform.
Data Pipeline Development: Design scalable ETL/ELT batch and streaming data processing pipelines using Python, PySpark, and SQL.
Performance Optimization: Troubleshoot and tune Spark jobs, memory allocation, and query execution for high-volume enterprise workloads.
Data Governance & Security: Implement Unity Catalog for access control, lineage, and data governance across multi-environment setups.
CI/CD & MLOps Integration: Establish and maintain automated CI/CD deployment pipelines (GitHub Actions, Azure DevOps, or AWS CodePipeline) for Databricks assets and Terraform configurations.
Collaboration: Work closely with data architects, software engineers, and analytics teams to translate business requirements into efficient technical architecture.

Required Qualifications & Skills
Databricks Expertise: 10+ years of hands-on experience in building enterprise data solutions on the Databricks platform (Delta Lake, PySpark, Unity Catalog, Databricks SQL).
Terraform Expertise: Mandatory experience writing, maintaining, and deploying Terraform modules specifically for Databricks resources and underlying cloud infrastructure.
Cloud Platforms: Solid hands-on experience in either Azure (Azure Databricks, ADF, ADLS Gen2) or AWS (S3, IAM, EMR).
Programming: Mastery of Python, PySpark, and advanced SQL.
Data Engineering Fundamentals: Strong background in data modeling, lakehouse architecture, real-time streaming, and batch ingestion.
DevOps & Automation: Experience with Git version control and CI/CD tools for automated code/infrastructure deployments.