Senior Databricks Architect

Job Title: Senior Databricks Architect

Location: Remote

Job Type: Long-Term Contract

Experience: 10+ years overall IT; 5+ years in Databricks / Cloud Data Engineering / Data Architecture

Job Overview

Key Responsibilities

• Define target-state Databricks Lakehouse (Medallion: Bronze/Silver/Gold) architecture
• Establish migration frameworks, design patterns, and coding standards
• Analyze and reverse-engineer existing Talend jobs, mappings, and dependencies
• Translate Talend transformations into PySpark/SQL/Databricks implementations
• Implement CDC, SCD Type X/X, data quality, error handling, and audit frameworks
• Optimize Spark workloads, Delta tables, partitioning, and job execution
• Design integrations with cloud storage, databases, messaging platforms, and APIs
• Provide technical leadership, mentoring, architecture reviews, and code reviews
• Communicate technical risks and recommendations to client leadership

Required Skills

• Databricks: Databricks, Apache Spark, PySpark, Delta Lake, Databricks Workflows, Unity Catalog, Databricks SQL, performance tuning
• Data Engineering: SQL, data modeling, ETL/ELT architecture, batch and incremental processing, CDC, SCD, data quality, large-volume data processing
• Talend: Talend Data Integration, job analysis, migration to cloud data platforms
• Cloud: Microsoft Azure or AWS, ADLS/S3, cloud networking, IAM/RBAC, secrets management
• DevOps: Git/GitHub/Azure DevOps, CI/CD, Terraform, automated testing, monitoring

Preferred Skills

• Databricks or Cloud certifications (Azure/AWS Data Engineer or Architect)
• Experience with Unity Catalog and data governance
• Kafka or other streaming technologies
• Azure Data Factory or equivalent orchestration tools
• Power BI or other BI platforms
• Retail/Grocery domain experience
• Experience with automated Talend-to-Databricks migration frameworks

Location & Work Model

Remote

Engagement Details

Domain: Retail/Grocery. This is a contract engagement. Education: Bachelor's or Master's in Computer Science, IT, Engineering, or related field preferred