Databricks Platform Administrator / Engineer

Position Details:

1. Job Title: Databricks Platform Administrator / Engineer
2. Location: 55 Water Street & Edition, NJ (Hybrid)
3. Duration: 12+ Months
4. Visa: / GC
5. Mode of Interview: Virtual

Job Summary:
The role involves managing, configuring, optimizing, and maintaining the Databricks environment to support data engineering, analytics, and machine learning workloads in a secure and compliant manner.

Key Responsibilities:

• Manage Databricks workspaces, clusters, jobs, libraries, and notebooks.
• Implement access controls using Unity Catalog, SCIM, and RBAC.
• Configure and monitor autoscaling, cluster policies, and compute usage.
• Enforce security policies and support compliance for P&C insurance.
• Build CI/CD pipelines for notebooks, jobs, and ML workflows using Azure DevOps, GitHub Actions, or Jenkins.
• Automate platform provisioning and configurations with Terraform/ARM.
• Set up monitoring, logging, and alerting using Azure Monitor, Datadog, or Splunk.
• Collaborate with Data Engineers, Data Scientists, and Business Analysts; provide training and documentation.

Required Skills & Experience:

• 5+ years in data platform engineering or system administration.
• 2+ years hands-on Databricks administration on Azure.
• Experience with Apache Spark, Delta Lake, Lakehouse architecture.
• Scripting/automation using Python, Bash, PowerShell, Terraform.
• Familiarity with data security, audit, and compliance frameworks (e.g., GDPR, NYDFS, SOC 2).
• Experience with Azure Data Services, ADLS, Key Vault, Azure DevOps.
• Strong understanding of P&C insurance data (policy, claims, underwriting, actuarial).

Preferred Qualifications:

• Databricks Certified Associate / Professional.
• Integration experience with Guidewire, Duck Creek, or similar P&C platforms.
• Knowledge of ACORD data models and ISO data feeds.
• Exposure to Unity Catalog, Purview, Collibra, Informatica.

Best Regards,

Hari Daile

Senior US IT Recruiter