Remote Data Architect with AI/ML, Databricks, Data Governance, Security, Model, ETL-C2C- Ch

Role: Data Architect

Location: Remote

Duration: 12+ Months

Note: Candidate must have Hands on experience in Databricks

Requirements:

1. Data Strategy & Architecture Development

• Define and implement the data architecture and data strategy aligned with business goals.
• Design scalable, cost-effective, and high-performance data solutions using Databricks on AWS, Azure, or Google Cloud Platform.
• Establish best practices for Lakehouse Architecture and Delta Lake for optimized data storage, processing, and analytics.

1. Data Engineering & Integration Architect ETL/ELT pipelines leveraging Databricks Spark, Delta Live Tables, and Databricks Workflows.

• Optimize data ingestion from sources like Oracle Fusion Middleware, Web Methods, MuleSoft, and Informatica into Databricks.
• Ensure real-time and batch data processing with Apache Spark and Delta Lake.
• Work on data integration strategies, ensuring seamless connectivity with enterprise systems (e.g., Salesforce, SAP, ERP, CRM).

1. Data Governance, Security & Compliance Implement data governance frameworks leveraging Unity Catalog for data lineage, metadata management, and access control.

• Ensure compliance with HIPAA, GDPR, and other regulatory standards in life sciences.
• Define RBAC (Role-Based Access Control) and enforce data security best practices using Databricks SQL and access policies.
• Enable data stewardship and ensure data cataloging for self-service data democratization.

1. Performance Optimization & Cost Management Optimize Databricks compute clusters (DBU usage) for cost efficiency and performance tuning.

• Define and implement query optimization techniques using Photon Engine, Adaptive Query Execution (AQE), and caching strategies.
• Monitor Databricks workspace health, job performance, and cost analytics.

1. AI/ML Enablement & Advanced Analytics Design and support ML pipelines leveraging Databricks ML flow for model tracking and deployment.

• Enable AI-driven analytics in genomics, drug discovery, and clinical data processing.
• Collaborate with data scientists to operationalize AI/ML models in Databricks.

1. Collaboration & Stakeholder Alignment Work with business teams, data engineers, AI/ML teams, and IT leadership to align data strategy with enterprise goals.

• Collaborate with platform vendors (Databricks, AWS, Azure, Google Cloud Platform, Informatica, Oracle, MuleSoft) for solution architecture and support.
• Provide technical leadership, conduct PoCs, and drive Databricks adoption across the organization.

1. Data Democratization & Self-Service Enablement Implement data sharing frameworks for self-service analytics using Databricks SQL and BI integrations (Power BI, Tableau).

• Promote data literacy and empower business users with self-service analytics.
• Establish data lineage and cataloging to improve data discoverability and governance.

1. Migration & Modernization Lead the migration of legacy data platforms (Informatica, Oracle, Hadoop, etc.) to Databricks Lakehouse.

• Design a roadmap for cloud modernization, ensuring seamless data transition with minimal disruption.

Mandatory Key Skills:

1. Databricks & Spark Expertise Strong knowledge of Databricks Lakehouse architecture (Delta Lake, Unity Catalog, Photon Engine).

• Expertise in Apache Spark (PySpark, Scala, SQL) for large-scale data processing.
• Experience with Databricks SQL and Delta Live Tables (DLT) for real-time and batch processing.
• Understanding of Databricks Workflows, Job Clusters, and Task Orchestration.

1. Cloud & Infrastructure Knowledge Hands-on experience with Databricks on AWS, Azure, or Google Cloud Platform (preferred AWS Databricks).

• Strong understanding of cloud storage (ADLS, S3, GCS) and cloud networking (VPC, IAM, Private Link).
• Experience with Infrastructure as Code (Terraform, ARM, CloudFormation) for Databricks setup.

1. Data Modeling & Architecture Expertise in data modeling (Dimensional, Star Schema, Snowflake, Data Vault).

• Experience with Lakehouse, Data Mesh, and Data Fabric architectures.
• Knowledge of data partitioning, indexing, caching, and query optimization.

1. ETL/ELT & Data Integration Experience designing scalable ETL/ELT pipelines using Databricks, Informatica, MuleSoft, or Apache NiFi.

• Strong knowledge of batch and streaming ingestion (Kafka, Kinesis, Event Hubs, Auto Loader).
• Expertise in Delta Lake & Change Data Capture (CDC) for real-time updates.

1. Data Governance & Security Deep understanding of Unity Catalog, RBAC, and ABAC for data access control.

• Experience with data lineage, metadata management, and compliance (HIPAA, GDPR, SOC 2).
• Strong skills in data encryption, masking, and role-based access control (RBAC).

1. Performance Optimization & Cost Management Ability to optimize Databricks clusters (DBU usage, Auto Scaling, Photon Engine) for cost efficiency.

• Knowledge of query tuning, caching, and performance profiling.
• Experience monitoring Databricks job performance using Ganglia, CloudWatch, or Azure Monitor.

1. AI/ML & Advanced Analytics

• Experience integrating Databricks ML flow for model tracking and deployment.
• Knowledge of AI-driven analytics, Genomics, and Drug Discovery in life sciences.

Thanks

Yashasvi Hasija

Technical Recruiter | Empower Professionals

......................................................................................................................................

| Phone: x 368 | Fax:

LinkedIn: linkedin.com/in/yashasvi-hasija-6a745625b

100 Franklin Square Drive Suite 104 | Somerset, NJ 08873

Certified NJ and NY Minority Business Enterprise (NMSDC)