Data Architect with Databricks

Role : Databricks Architect

Location : Iselin, NJ (Hybrid)

Duration: Long Term Project  

Responsibilities:

·       Define and maintain reference architectures (Lakehouse, CDC, streaming) and domain data models (conceptual, logical, physical).

·       Create and enforce data standards: naming conventions, data types, modeling practices, semantic definitions (aligned to business glossaries).

·       Establish metadata operating model: ownership, stewardship, processes for Catalog, Glossary, Data Dictionary, and Data Lineage.

·       Integrate lineage capture across pipelines (ETL/ELT/streaming), BI layers, and ML workflows.

·       Architect cross-platform data flows across Databricks, Oracle/SQL Server, Snowflake and metadata tools.

·       Define IAM models: RBAC/ABAC, SSO/federation, SCIM provisioning; directory-driven entitlements and periodic access reviews.

·       Define catalog strategy (e.g., Unity Catalog/Purview/Collibra/Alation) and integrate with CI/CD for automated registration and lineage.

·       Design reusable pipeline frameworks with configuration-driven IO, logging, metrics, retry/error handling, and data quality checks.

Skills:

Data Modeling: Dimensional (star/snowflake), 3NF, Data Vault, business glossary-to-model mapping, SCD types, time-series/event modeling.

Metadata & Governance: Practical use and integration of Data Catalogs, Lineage,

Oracle/SQL Server (data modeling, migration/CDC patterns).

Snowflake (roles, warehouses, performance tuning, tasks/streams, dynamic tables).

Databricks/Spark (SQL/PySpark, Structured Streaming, Delta Lake; Unity Catalog).

Security & Compliance: IAM/RBAC, masking, tokenization, encryption; PCI/AML/KYC/GDPR/DPDP awareness.

Integration & Orchestration: Databricks Workflows, Airflow/ADF, API integrations with catalog tools; schema registry

Exceptional interpersonal and collaboration skills within a team environment