SR Azure DataBricks Engineer

Position: Sr Azure Data Bricks Engineer

Location: Dallas,TX(Onsite)

OVERALL EXPEREINCE 10YRS+

Job Description:

Prepare the Databricks Serverless environment for the team

Set up job clusters

Ensure all libraries needed are present

Work with Databricks and CDO to ensure all AI capabilities are enabled

Migrate existing Databricks jobs and workflows

Resolve Azure networking, proxy, firewall, and authentication issues in Databricks and web applications

Optimize performance, cost, and scalability of Databricks workloads

Enhance monitoring, alerting, and job reliability improvements by creating a UI that tracks all jobs start, completion, duration and status

Maintain and enhance DataSphere health check framework

Support Unity Catalog, access controls, and environment governance

Troubleshoot production issues and failed jobs

Create Genie Spaces for each Domain as the jobs are migrated

Strong Databricks experience, including workflows, jobs, clusters, serverless, and performance optimization

Azure cloud experience, including networking, proxy, firewall, authentication, and application deployments

Python, SQL, ETL development, and migration experience with Databricks and Snowflake

Production data pipeline support, monitoring, alerting, and job reliability

Unity Catalog, access controls, governance, and data quality frameworks

At least one contractor must have React, TypeScript/JavaScript, API integration, and custom Azure web app development experience

Preferred: 
Databricks AI capabilities, AI/LLM applications, health check automation, dashboards, SSO/OAuth/Entra ID, and operational UI development