Senior Infrastructure Engineer – Terraform & Databricks

Hybrid in San Francisco, CA, US • Posted 18 hours ago • Updated 18 hours ago
Contract Independent
Contract Corp To Corp
Contract W2
1 Year
No Travel Required
Hybrid
Depends on Experience
Fitment

Dice Job Match Score™

🔢 Crunching numbers...

Job Details

Skills

  • DevOps engineering
  • Databricks on AWS

Summary

Senior Infrastructure Engineer – Terraform & Databricks

San Francisco, CA

Type : Contract

Need candidates who can work onsite from Day 1 (3 days a week : Tuesday, Wednesday and Thursday – 3 days work from office) – NO REMOTE

Highly preferred local candidates

 

Experience: 09+ years

About the role

Building an enterprise Databricks Lakehouse on AWS, and all of it is managed as code.  In this role you’ll design, build and run the Terraform/Terragrunt foundation behind our Databricks workspaces, Unity Catalog, compute and security controls.

You’ll work closely with Cloud Infrastructure/SRE, Security Engineering and Data Engineering. Your goal is to take our platform from sandbox to production-ready multi-environment setup that is secure, well governed and cost-efficient.

What you will do

  • Design, build and maintain reusable Terraform modules for Databricks account-level (MWS) and workspace-level resources. This covers workspaces, Unity Catalog (metastores, catalogs, schemas, volumes, grants), cluster policies, instance pools, SQL warehouses, jobs, DLT pipelines and service principals.
  • Use Terragrunt to manage environments consistently across sandbox, nonprod, preprod and prod, with minimal configuration drift.
  • Provision and secure the supporting AWS infrastructure: customer-managed VPCs, PrivateLink/VPC endpoints, security groups, NACLs, IAM roles and policies, S3, KMS customer-managed keys (CMK) and Secrets Manager.
  • Lead security hardening. That means fixing findings from security architecture reviews and cloud security posture management (CSPM) tools such as Wiz, enforcing PrivateLink-only access, IP access lists, CMK encryption and secret/OAuth token rotation.
  • Set up identify and access controls, including Okta SSO and SCIM sync, role-based (RBAC) and tag-based (ABAC) access in Unity Catalog, and least-privilege service principals.
  • Build and own CI/CD for infrastructure. This includes GitHub Actions plan/apply workflows with approvals, pre-commit hooks, Terratest, and security scanning with Checkov or tfsec.
  • Set up monitoring and alerting with CloudWatch, Grafana and Splunk log forwarding. That covers cluster and job health, token expiry and cost anomalies.
  • Drive FinOps practices: cluster policy guardrails, spot usage, moving work-loads from all-purpose to job clusters, enforcing tagging, and analysing DBU usage.
  • Enable integrations with Dagster, dbt Cloud and BI tools such as Tableau, Power BI and Looker via SQL warehouses.
  • Write and maintain architecture docs, ADRs, runbooks and module references, and enforce naming and tagging standards.
  • Databricks – Hands-on build data pipelines using Pyspark
  • Used Agentic IDE to build data pipelines
  • Understands Medallion architecture
  • Understands ML flow – Model inferencing
  • Have domain expertise or understanding of Credit decisioning or Risk or Servicing (collections)

Skills Required

  • 7+ years in infrastructure, platform or DevOps engineering with atleast 3 years of hands-on Terraform in production.  Terragrunt experience is strongly preferred.
  • 2+ years running Databricks on AWS, including workspace provisioning, Unity Catalog, cluster policies and the Databricks Terraform provider.
  • Deep AWS knowledge; VPC networking, PrivateLink, IAM, KMS, S3, Secrets Manager and CloudWatch.
  • Experience with CI/CD for Infrastructure as code (GitHub Actions, Jenkins or similar) and with GitOps workflows.
  • Strong grasp of cloud security and compliance: encryption, network isolation, least privilege and audit logging. Experience in regulated environments such as financial services is a plus.
  • Scripting skills in Python and/or Bash. Go is a plus for Terratest.
  • Comfortable with Git, Code reviews and writing clear technical documentation.

Nice to have

  • Databricks certifications (Data Engineer Associate/Professional or Platform Administrator), or HashiCorp Terraform Associate.
  • Experience with Dela Live Tables, Vector Search, Model Serving or MLops Infrastructure.
  • Familiarilty with Dagster, Dbt, Spark and Delta Lake.
  • Experience with Wiz, Checkov, tfsec or similar policy-as-code and CSPM tools.
  • Background in FinOps or cloud cost optimization.
  • Experience with SOC 2 or similar compliance frameworks
Tekshapers is an equal opportunity employer and will consider all applications without regards to race, sex, age, color, religion, national origin, veteran status, disability, sexual orientation, gender identity, genetic information or any characteristic protected by law.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: 10335759
  • Position Id: 30732-17456-1794862302
  • Posted 18 hours ago
Contact the job poster
JP

Job Poster

Recruiter @ TekShapers

Create job alert
Never miss an opportunity! Create an alert based on the job you applied for.