Lead Databricks Unity Catalog Engineer

 Lead Databricks Unity Catalog Engineer - (Azure DevOps, GitHub, Atlassian, Jira)

Job Type: Contract (6-12 Months)

Location: Philadelphia, PA - hybrid model

Job description:

Job Summary

• We are looking for an experienced Lead  or Developer with strong expertise in Unity Catalog,  data governance, and enterprise data platform management.
• The ideal candidate should have hands-on experience with Unity Catalog, access control, metadata management, data lineage, and governance within a Databricks environment.
• The candidate will lead data governance initiatives, manage platform integrations, and collaborate with data engineering, DevOps, security, and business teams to establish scalable and secure data management practices.
• Experience with Azure DevOps, GitHub, Atlassian, and Jira is required.
• Experience in Wealth Management or Financial Services is highly preferred.

Key Responsibilities

• Lead the implementation, administration, and management of Databricks Unity Catalog.
• Establish and manage data governance policies, access controls, permissions, and security standards.
• Implement and maintain data lineage, metadata management, and data discovery processes.
• Manage Unity Catalog integration with Databricks workspaces and enterprise data platforms.
• Work with Azure DevOps and GitHub to manage CI/CD pipelines, version control, and deployment processes.
• Establish and manage development workflows using GitHub and Azure DevOps.
• Utilize Jira and Atlassian tools for project tracking, sprint planning, task management, and team collaboration.
• Collaborate with Data Engineering, Cloud, Security, and Architecture teams to implement governance standards.
• Define best practices for data access management, auditability, and compliance.
• Troubleshoot platform, deployment, access, and integration-related issues.
• Lead technical discussions, mentor team members, and coordinate with stakeholders.
• Ensure data governance practices align with enterprise security and regulatory requirements.

Required Qualifications

• 8+ years of experience in Data Engineering, Data Platform Management, or related technical roles.
• Strong hands-on experience with Databricks Unity Catalog.
• In-depth understanding of data governance, metadata management, data lineage, and access controls.
• Experience with role-based access control (RBAC), permissions, and data security.
• Hands-on experience with Azure DevOps and GitHub.
• Experience managing CI/CD pipelines, Git repositories, and deployment workflows.
• Working experience with Atlassian tools, particularly Jira.
• Strong understanding of cloud-based data platforms and enterprise data architecture.
• Excellent leadership, communication, problem-solving, and stakeholder management skills.

Preferred Qualifications

• Experience with Azure Databricks and Microsoft Azure cloud services.
• Knowledge of Delta Lake, Apache Spark, and lakehouse architecture.
• Experience implementing enterprise data governance frameworks.
• Understanding of financial services, wealth management, and regulatory compliance requirements.
• Experience leading technical teams and enterprise data platform initiatives.