Solution Databricks Architect(20 hours/Week, Part-time)

Type: Remote

Work Hours : 20 hours (4 hrs a day)

Role Overview

We are looking for an experienced Databricks Architect to support a large-scale data modernization initiative for Client. The consultant will be responsible for designing scalable data architecture, defining Databricks best practices, and leading migration and implementation activities across the enterprise data platform.

The ideal candidate should have strong hands-on experience with Databricks, Apache Spark, Python/PySpark, Delta Lake, Unity Catalog, cloud data platforms, and enterprise data architecture.

Key Responsibilities

• Design and implement enterprise-scale Databricks architecture for complex data engineering and analytics workloads.
• Define end-to-end architecture covering data ingestion, transformation, storage, governance, security, orchestration, and consumption.
• Lead migration of legacy/on-premise data workloads to a modern cloud + Databricks platform.
• Design and implement Medallion Architecture – Bronze, Silver, and Gold layers.
• Build and optimize large-scale data pipelines using Python, PySpark, Spark SQL, and Delta Lake.
• Define standards for Delta Tables, schema evolution, partitioning, OPTIMIZE, Z-Ordering, caching, and performance tuning.
• Architect and implement Unity Catalog for centralized governance, access control, lineage, and data security.
• Establish appropriate RBAC, service principals, secrets management, and data-access policies.
• Design batch and, where required, streaming data processing solutions.
• Provide architectural guidance around Spark execution, cluster configuration, memory management, shuffle optimization, and performance troubleshooting.
• Design integration patterns between Databricks and enterprise systems, APIs, databases, data warehouses, and cloud storage.
• Establish CI/CD and DevOps standards for Databricks notebooks, workflows, jobs, and infrastructure deployments.
• Define development standards across Dev, QA, UAT, and Production environments.
• Conduct architecture reviews and provide technical leadership to Data Engineers and Databricks developers.
• Work closely with enterprise architects, business stakeholders, security teams, data governance teams, and application teams.
• Translate business and technical requirements into scalable architecture and implementation plans.
• Support technical assessments, POCs, design documentation, and solution architecture presentations.
• Identify performance, scalability, security, and operational risks and recommend appropriate solutions.
• Provide technical ownership from discovery and architecture through implementation and production deployment.

Required Skills

Databricks

• 8+ years of overall Data Engineering / Data Platform experience.
• Strong hands-on experience with Azure Databricks or Databricks on AWS.
• Multiple enterprise-level end-to-end Databricks implementations.
• Strong understanding of Databricks platform architecture and administration.
• Hands-on expertise with:
• Delta Lake
• Delta Tables
• Unity Catalog
• Databricks Workflows / Jobs
• Auto Loader
• Databricks SQL
• Cluster configuration and optimization
• Performance tuning

Spark / PySpark

• Advanced knowledge of Apache Spark and PySpark.
• Strong understanding of Spark internals including:
• Driver and Executors
• DAG and stages
• Lazy evaluation
• Shuffle operations
• Partitioning
• Repartition vs. Coalesce
• Broadcast joins
• Data skew
• Memory management
• Spark performance optimization

Data Architecture

• Strong experience designing enterprise data platforms.
• Expertise in Medallion Architecture.
• Strong understanding of:
• Data lakes
• Lakehouse architecture
• Data warehouses
• Data modeling
• ETL/ELT
• Batch processing
• Data quality
• Metadata management
• Data lineage

Programming

• Advanced Python/PySpark development experience.
• Strong SQL skills.
• Ability to write and review production-quality data engineering code.

Cloud

Strong experience with at least one major cloud platform:

• AWS: S3, IAM, Glue, Lambda, Redshift, Secrets Manager, CloudWatch, etc.

OR

• Azure: ADLS Gen2, ADF, Key Vault, Azure DevOps, Entra ID, etc.

Exposure to multiple cloud platforms would be preferred.

Security & Governance

• Strong experience with Unity Catalog.
• Data access control and RBAC.
• Data lineage.
• PII/sensitive-data handling.
• Secrets and credential management.
• Enterprise data-governance practices.

DevOps / CI-CD

• Experience implementing CI/CD for Databricks solutions.
• Git-based development.
• Azure DevOps / GitHub / Jenkins or similar tools.
• Automated deployment across multiple environments.
• Infrastructure-as-Code exposure is preferred.

Preferred Qualifications

• Databricks Certified Data Engineer Professional or Databricks Architect-level certification.
• Experience within Insurance, Financial Services, or highly regulated enterprise environments.
• Experience modernizing large legacy data ecosystems.
• Experience working directly with US-based enterprise stakeholders.
• Knowledge of MLOps, AI/ML, or GenAI workloads on Databricks is an advantage.
• Experience designing reusable frameworks and accelerators for enterprise data engineering.

Ideal Candidate Profile

The ideal consultant should be able to operate as both a hands-on Databricks expert and Solution Architect. The person should not be limited to notebook development; they should be capable of understanding the complete enterprise landscape, making architectural decisions, defining platform standards, guiding engineering teams, and taking responsibility for successful end-to-end delivery.

Key Evaluation Parameters

• Primary Focus: Databricks architecture, modernization, migration, governance, performance, and enterprise data engineering