Data Architect (with OpenShift - OCP)

Project: Identity & Access Management (IAM) Data Modernization

Migration of an on-premises SQL data warehouse to a modern enterprise Data Lake platform, enabling analytics and GenAI use cases. The platform leverages PySpark-based processing, CI/CD pipelines, and containerized deployments on OpenShift (OCP), with Google Cloud Platform as a preferred cloud platform, to deliver scalable, secure, and high-performance data solutions

About Program/Project

The IAM Data Modernization program focuses on transforming legacy data platforms into a scalable and cloud-compatible architecture.

Key Highlights:

• Integration Scope: 30+ source systems with multiple downstream integrations
• Capabilities: Metrics, reporting, advanced analytics, and GenAI use cases (NL querying, summarisation, cross-domain insights)
• Benefits:
• Scalable and resilient data platform
• High-performance semantic and analytics layer
• Single source of truth for enterprise-wide reporting and analytics

Role Summary

We are looking for a Data Architect with strong expertise in OpenShift (OCP), PySpark, and CI/CD pipelines to design and govern scalable data platforms.

The role requires defining end-to-end data architecture, containerised deployment patterns, orchestration strategies (Airflow/Autosys), and platform standards, along with hands-on involvement in implementation.

Key Responsibilities

Data Architecture & Platform Design

• Define enterprise data architecture for IAM data lake and analytics platform
• Design scalable, modular, and containerised data pipeline architectures on OCP
• Establish data models, schema governance, and data lifecycle strategies
• Define best practices for data partitioning, performance optimisation, and cost efficiency

OpenShift (OCP) & Platform Engineering

• Architect and govern containerised data workloads on OpenShift (OCP)
• Define standards for deployment, scaling, and workload isolation
• Collaborate with DevOps teams for platform engineering and infrastructure alignment

Big Data & Processing (PySpark Focus)

• Define architecture for PySpark-based batch and near real-time processing pipelines
• Provide guidance on distributed processing design, optimisation, and performance tuning
• Establish reusable frameworks for ETL/ELT processing

Data Ingestion & Orchestration

• Architect data ingestion frameworks (batch, streaming, CDC)
• Define orchestration strategies using Airflow / Autosys
• Implement standards for retry, backfills, dependency management, and error handling

DevOps / CI-CD

• Define and oversee CI/CD strategy for data and platform deployments
• Enable automation of build, test, and deployment processes
• Ensure integration of CI/CD pipelines with OCP-based environments

Cloud & Data Platforms (Preferred)

• Provide architecture guidance for Google Cloud Platform-based data platforms (preferred, not mandatory)
• Define integration patterns for cloud-native and on-premise hybrid environments
• Guide teams on cloud migration strategies and modern data platform adoption

Data Governance, Quality & Observability

• Define frameworks for:
• Data quality, validation, and lineage
• Metadata management and cataloguing
• Establish monitoring, logging, alerting, and SLOs for platform reliability
• Ensure compliance with data security and audit requirements

Stakeholder Collaboration

• Work closely with client architects, IAM teams, and business stakeholders
• Translate business requirements into scalable technical architecture
• Provide architectural guidance and mentorship to engineering teams

Required Skills

Core Skills (Must Have)

• Strong experience in:
• OpenShift (OCP) / Kubernetes-based platforms
• PySpark / Spark ecosystem
• CI/CD implementation for data platforms
• Airflow / Autosys orchestration tools
• Solid understanding of:
• Data lake architectures (layered models)
• ETL/ELT design patterns
• Distributed data processing concepts

Data Engineering & Storage

• Expertise in:
• Data formats: Parquet, ORC, Avro
• Partitioning and performance tuning
• Large-scale data modelling for analytics

Cloud (Preferred Not Mandatory)

• Experience with Google Cloud Platform (Google Cloud Platform) (preferred)
• Exposure to services like BigQuery, Dataproc, Dataflow, GCS is a plus

Observability & Reliability

• Experience defining:
• Monitoring, logging, alerting frameworks
• Dashboards, SLOs, and operational runbooks

Good to Have

• Experience with IAM domain / cybersecurity data
• Understanding of data security and access control frameworks
• Exposure to GenAI-enabled data platforms
• Experience in Agile delivery and team leadership

Qualifications

• Experience:
• 10 14+ years in Data Architecture / Data Engineering
• Strong experience in OCP, PySpark, CI/CD, and orchestration frameworks
• Prior experience in data modernisation / migration programs
• Education:
  Bachelor s/Master s in Computer Science, Information Systems, or equivalent
• Certifications (Preferred):
• OpenShift / Kubernetes certifications
• Google Cloud Platform certifications (preferred, not mandatory)