Position:Databricks Platform Engineer
Location:Bloomington,IL ( 90% offsite, 10% onsite)
Duration:Long term
Job Description:
** Do not submit profiles limited to traditional ETL tools, basic SQL development, or AWS Glue without meaningful hands-on Databricks and Spark experience.
Databricks Platform Engineer: Must-Have Skills
6+ years of cloud platform or data infrastructure engineering experience, including 2+ years of production Databricks experience.
Strong hands-on experience designing and administering Databricks workspaces on AWS.
Deep AWS infrastructure experience with VPC, IAM roles and policies, cross-account access, SCPs, S3, KMS, Secrets Manager, and CloudTrail.
Strong Terraform experience, specifically with the Databricks Terraform provider.
Hands-on expertise with Unity Catalog administration, including metastore configuration, catalog and schema ownership, storage credentials, external locations, privilege management, and RBAC.
Experience implementing column-level masking, row-level security, dynamic views, lineage, and audit logging.
Strong knowledge of private networking, including AWS PrivateLink, VPC peering, transit gateways, security groups, NACLs, and no-public-IP configurations.
Hands-on experience defining Databricks cluster policies, instance pools, job clusters, autoscaling, Photon, runtime standards, and cost controls.
Experience with Databricks Workflows, system tables, CLI, SDK, Repos, and Git integration.
Strong CI/CD experience using GitHub Actions, Azure DevOps, or equivalent for Databricks deployments.
Experience implementing platform monitoring, cost attribution, and observability using Databricks system tables, Datadog, Grafana, or equivalent tools.
Ability to create platform architecture documentation, operational runbooks, onboarding standards, and production support procedures.
Profiles should demonstrate all the following:
Production Databricks platform administration on AWS
Unity Catalog implementation and administration
Terraform-based Databricks provisioning
AWS networking and cross-account IAM experience
Databricks security, governance, cluster policies, and cost controls
CI/CD automation for Databricks platform resources
Thanks,
Ganesh R
Jupiter Consulting Services
MWBE Certified Company
16024 Manchester Road, Suite 262, St. Louis, MO 63011.
Email
Jupiter Consulting Services, LLC is an Equal Opportunity Employer (EOE).
