Sr. Data bricks Architect(Can be remote)

We are seeking a highly skilled Sr. Data bricks Architect with experience in data architecture and engineering. The ideal candidate will have a strong background in designing and implementing large-scale data platforms, particularly using Data bricks. This role requires technical leadership, strategic thinking, and the ability to collaborate with cross-functional teams to drive data initiatives.

Key Responsibilities

Technical Leadership

• Provide technical and people leadership to design and build lakehouse and analytic products.
• Serve as a key member of the Client Architecture Review Board and Design Steering Committee, sharing expertise on architectures and design initiatives.

Data Architecture Design

• Involved in the design and implementation of large-scale platform and data architectures on Databricks.
• Develop technical roadmaps for data architecture, aligning with business objectives and industry trends.
• Design end-to-end data management and segmentation for the entire data flow in the Databricks platform.

Migration and Catalog Management

• Lead the migration from Hive Metastore to Unity Catalog to enable trickle-down catalog for development, staging, and production environments.
• Build different user personas to cater to wider consumption patterns in Unity Catalog.
• Implement a unified access control model on Databricks' Unity Catalog, leveraging RBAC, ABAC, and CBAC for fine-grained data governance.

Data Governance and Compliance

• Implement Immuta and Databricks integration for automatic financial and PII data discovery, ensuring comprehensive data governance.
• Design and build a comprehensive data quality framework and maintain statistics for further analysis.

Data Pipeline Development

• Design and build complex data pipelines for both batch and streaming data sources on Databricks using Auto Loader, Kafka, Airflow, and workflows.
• Implement Delta Sharing to enable easy and secure data sharing within and outside the organization.

Hybrid Data Solutions

• Implement data virtualization to provide a unified data layer across hybrid environments.
• Design and build lakehouse federated queries and push-down queries as hybrid compute for external CDP platforms.

Infrastructure Management

• Build a Terraform-based Databricks IaaS framework to support workspace management, IAM, SSAM, service principal, and other operations.
• Create a robust DevOps framework for Databricks platform and data pipelines.