Senior Data & AI Platform Architect/Engineer

Overview

We are seeking an experienced Senior Data & AI Platform Architect (Contractor) to design, build, and deliver an enterprise-grade, secure, end-to-end Data & AI Platform. The contractor will implement ingestion, secure data transfer, object storage, governance, ML engineering, metadata, observability, and AI security layers across the full modern data stack based on Databricks over a 9 month roadmap.

The ideal candidate has deep hands-on experience with Databricks, DataHub , Progress semaphore, GitLab CI/CD , MinIO/Alluxio storage solution, Ignition,HashiCorp Vault,  Splunk,  PointGuard AI, and large-scale data architectures.

Responsibilities

·      Architecture & Platform Deployment/Development

·      Architect and implement a full-scale, cloud-native Data & AI platform based on Databricks and Datahub integrating ingestion, storage, processing, ML, security, and governance components.

·      Design and deploy Ignition-based ingestion/orchestration pipelines for batch, streaming, and event-driven data.

·      Implement secure, compliant data transfer mechanisms (accelerated transfer, API-based ingestion, encryption in transit).

·      Deploy and operationalize storage/iceberg table format storage as the core object store integrated with Databricks and feature pipelines.

·      Build scalable ETL/ELT data pipelines using Databricks, Delta Lake, Unity Catalog, and Lakehouse best practices.

Machine Learning & MLOps

·      Deploy Databricks Feature Store, MLflow Model Registry, and automated ML pipelines.

·      Implement model governance, lineage, monitoring, rollback, and lifecycle management.

·      Deliver end-to-end pipelines for both traditional ML and LLM/GenAI (Agent Bricks, RAG, model serving).

Security & Compliance

·      Integrate PointGuard AI for AI posture, anomaly detection, compliance, and secure model operations.

·      Implement HashiCorp Vault for secrets management, transit encryption, and certificate workflows.

·      Build zero-trust RBAC/ABAC frameworks aligned with enterprise and federal standards.

·      Integrate audit logging to Splunk across all platform components.

Governance & Metadata

·      Deploy and configure DataHub for metadata ingestion, data lineage, dataset classification, and lifecycle management.

·      Integrate DataHub with Databricks, Vault, storage solution and orchestration pipelines.

·      Implement governance policies including data classification, tagging, and access workflows.

CI/CD & DevOps

·      Build and maintain GitLab CI/CD pipelines for data pipelines, ML pipelines, IaC, and model deployments.

·      Implement automated code scanning, quality gates, DAG submission workflows, and artifact management.

·      Use Terraform for IaC, resource deployment, and environment reproducibility.

Observability & Monitoring

·      Design and implement observability dashboards for data pipelines, ML pipelines, and infrastructure components.

·      Leverage Datadog with Databricks jobs,cluster,workflow etc monitoring.

·      Integrate real-time audit, compliance logs, and anomaly alerts into Splunk.

Collaboration & Delivery

·       Work closely with internal engineering team, data governance, and security teams.

·       Provide knowledge transfer, documentation, and hands-on training.

·       Deliver project milestones on a structured 9-month roadmap.

Required Qualifications

Technical Skills

• 10+ years building enterprise Data platforms, ML pipelines, or cloud-native architectures
• Strong hands-on experience with:
  
  

o   Databricks (Delta Lake, Workflows, Feature Store, MLflow, Model Serving, Unity Catalog, Declarative pipeline, Clean Room)

o   Ingestion and Orchestration -Prefect (or Apache Airflow/Dagster)

o   Familiarity with MinIO Aistor object storage  or Alluxio and secure Data transfer and Data lineage

o   HashiCorp Vault secrets and AWS KMS for secret management(Experience connecting Vault to Databricks & CI/CD pipelines)

o   GitLab CI/CD pipelines (Pipeline-as-code,DAG submission automation,Automated code scanning (SAST/DAST),Infrastructure-as-code (Terraform)

o   ML model deployment automation

o   Splunk observability integration (logs from storage access, Vault, Databricks, orchestration engines etc)

o   Ability to integrate Datahub with Databricks and ML ecosystem (Automated dataset classification,lineage tracking etc)

o   PointGuard AI or similar AI/security posture/runtime tools,AI model risk management framework,RBAC/ABAC architecture

·       Proficiency in Python, SPARK, SQL, REST APIs,Terraform

·       Experience with Kubernetes, Docker

·       Cloud expertise in AWS/Azure/Google Cloud Platform (AWS preferred)

·       Strong understanding of distributed systems, security architecture, governance, and data compliance

Preferred Qualifications

·       Databricks Certified Data Engineer Professional or Databricks Certified Machine learning Professional.

·       Experience implementing AI governance frameworks (NIST AI RMF, DOE cybersecurity frameworks)

·       Experience in public sector, government, or regulated environments

·       Experience with homomorphic encryption or advanced secure computation

·       Experience integrating Denodo or similar data virtualization layers (optional)

Soft Skills

·       Excellent documentation and diagramming skills

·       Ability to translate business requirements into technical designs and technical execution

·       Strong communication skills with technical & non-technical stakeholders

·       Ability to work independently and deliver long-term projects with minimal oversight

Duration & Engagement

·       9-month contract with possible extension(01/12/2026-09/30/2026)

·       Full-time engagement -40 hours per week

·       Hybrid with few days onsite depending on requirements