Databrick Architect

Role : Databrick Architect

Location : Tarrytown New York (Onsite)

Key Responsibilities & Skills

·        Lead enterprise-scale data platform modernization initiatives, driving migration from AWS EMR, Apache NiFi, and legacy ETL frameworks to the Databricks Lakehouse Platform. 

·        Architect and implement scalable Lakehouse solutions using Databricks, Delta Lake, Unity Catalog, and Databricks Workflows. 

·        Design and govern end-to-end data pipelines for batch, streaming, CDC, and real-time data integration workloads.

·        Provide architecture leadership for large-scale AWS-based data ecosystems leveraging S3, IAM, Redshift, Glue Catalog, Airflow, and Databricks.

·        Develop enterprise data architecture standards covering data modelling, metadata management, lineage, governance, security, and compliance.

·        Drive adoption of Databricks best practices including Delta Live Tables (DLT), Auto Loader, Unity Catalog, Serverless Compute, Lakehouse Federation, and advanced optimization techniques. 

·        Lead replatforming and migration programmes involving PySpark applications, Airflow DAGs, EMR workloads, Redshift integrations, and NiFi pipelines.

·        Partner closely with business stakeholders, enterprise architects, product owners, and data science teams to define technology roadmaps and target architectures.

·        Establish governance frameworks using Unity Catalog, data quality controls, observability, monitoring, and operational excellence practices

·        Provide technical leadership, mentoring, architecture reviews, design governance, and solution sign-offs across multiple delivery teams.

·        Lead architecture workshops, executive presentations, solution assessments, and technology evaluations.

Mandatory Technical Skills

·        Databricks Lakehouse Platform

·        Delta Lake, Delta Live Tables (DLT)

·        Unity Catalog

·        Databricks Workflows

·        Auto Loader

·        PySpark, Spark SQL, Python

·        AWS (S3, IAM, Glue, Redshift, EMR, Lambda)

·        Apache Airflow

·        CDC & Data Migration Frameworks

·        Data Governance & Security

·        CI/CD, GitHub, Jenkins

JD:

·        Lead enterprise-scale Databricks Lakehouse Architecture design and implementation on AWS.

·        Drive large-scale data platform modernisation and cloud transformation initiatives.

·        Architect scalable Medallion Architecture (Bronze, Silver, Gold) data platforms.

·        Lead migration of legacy EMR, NiFi, Redshift, and ETL workloads to Databricks.

·        Design high-performance batch and real-time data processing solutions.

·        Build robust ingestion frameworks using Auto Loader, Delta Lake, and Structured Streaming.

·        Define enterprise data governance and security standards using Unity Catalog.

·        Architect metadata-driven and reusable PySpark-based ETL/ELT frameworks.

·        Establish best practices for performance tuning, scalability, and cost optimisation.

·        Design and implement data quality, lineage, and observability frameworks.

·        Drive adoption of CI/CD, DevOps, Infrastructure as Code, and automation practices.

·        Collaborate with business, analytics, and engineering teams to define target-state architectures.

·        Conduct architecture reviews and provide technical leadership across multiple projects.

·        Mentor architects and senior engineers on Databricks and AWS best practices.

·        Design secure and scalable solutions leveraging AWS services (S3, Glue, Athena, Lambda, IAM, Redshift).

·        Implement and govern enterprise-wide data access, compliance, and security controls.

·        Evaluate and adopt latest Databricks capabilities such as DLT, Lakehouse Federation, Serverless, and MLflow.

·        Enable AI/ML, Generative AI, and advanced analytics use cases on the Lakehouse platform.

·        Create architecture roadmaps, migration strategies, standards, and governance frameworks.

·        Act as the primary technical advisor for customer leadership on data strategy and platform evolution.