Senior DevOps Platform Engineer/Platform Architect

Position: Senior DevOps Platform Engineer/Platform Architect

Location: Remote

Only Local Profiles to NJ

• The Platform Architect is a key member of the BSI Platform/Enterprise Data Platform team responsible for designing, deploying, and maintaining the foundational infrastructure components in AWS and Databricks. In addition to DevOps and security responsibilities, this role focuses on creating scalable, maintainable solutions via IaC.
• The Platform Architect ensures BCBSA BSI Platform/Data Platform environment adheres to regulatory requirements while adding new development.

Cloud Infrastructure (AWS) Architecture & Deployment

• Design and implement AWS infrastructure using AWS CDK, and Databricks using Terraform. Evaluate and optimize cloud resources for cost, performance, and security.

Databricks Administration & Architecture

• Provision and configure Databricks workspaces, clusters, and jobs. Develop and enforce governance policies, security controls, and permissions management within Databricks.

Devops & Automation

• Build and maintain CICD pipelines. Implement version control, PRs, automated testing.

Security & Compliance

• Integrate security controls to meet or exceed SOC2 requirements. Implement logging, monitoring, and alerting solutions to ensure compliance and operational excellence.

Collaboration & Technical Leadership

• Work closely with data engineers and cloud security teams to align on evolving requirements.
• Propose platform optimizations, implement code, and respond to incident response as required.

Documentation & Knowledge Sharing

• Produce and maintain technical documentation of technical components and development practices.

Responsibilities:

• Cloud Infrastructure & Platform Ownership (AWS + Databricks)

AWS Environment Management

• Account and network management/configuration management, security groups, Roles, Bucket Policies, other cloud infrastructure processes required for monitoring, maintenance, and ongoing development
• Using AWS CDK to define and deploy infrastructure (S3 Buckets, MWAA/Airflow environments, Terraform Private Agents + ECR and other containerized services)

Databricks Administration

• Workspace creation, cluster policies, job configurations, and managing tokens and service principals and other authorizing and authenticating technologies (SAML/SSO/Okta).
• Integration of Databricks with AWS services (S3, IAM, KMS etc).
• Development of Enterprise level data platform enablement, end user enablement, and data maturation across all BCBSA.

Infrastructure as Code (IaC)

• Maintaining AWS components via AWS CDK and AWS CLI.
• Deploying and configuring Databricks resources using Terraform.
• Code reviews, version control, CICD via SbD.

Data Engineering & Orchestration

Data Ingestion Pipelines

• Managing data ingest from external sources into S3
• Designing and maintaining Airflow DAGs in MWAA to trigger Databricks jobs
• Ensuring data ingestion meets performance, cost, security requirements.

Transformation & Analytics Enablement

• Building out data transformation and data serving technologies for BCBSA enterprise
• Enabling advanced analytics workloads (e.g. collaborative notebooks, ML experiments, near-real-time dashboards, etc)

Monitoring & Observability

• Setting up logs, metrics, and alerts for pipelines and job executions.
• Using CloudWatch and Databricks system audit logs for real-time performance visibility.

DevOps & Code Ownership Git Code Review & Approval

• Acting as code owners for all data platform repositories.
• Reviewing Pull Requests (PRs), ensuring adherence to coding standards, best practices, and compliance.

CICD Pipelines

• Maintain and operate BSI CICD SbD application stack.
• Includes promotion of Databricks notebooks and other data transformation or data serving code.

Automation & Tooling

• Developing custom automation scripts for deployment and maintenance.
• Monitoring code scans and environment scans for whole of BSI and MDP footprint. SOC2 Program Ownership for BSI/NDW/MMI/VBP/Portal entire NDW stack on BSI and BSI itself, and enterprise enablement

SOC2 Governance for BCBSA Cloud Applications

• Designing and implementing processes to automatically collect audit and compliance evidence both for Data Platform technology (BSI) but as well as other BCBSA cloud applications undergoing compliance review.
• Integrating with source systems to gather logs, configuration data, and access records.

Control Implementation & Validation

• Implementing technical controls (encryption, access controls, etc) required by SOC2.
• Regularly validating these controls, e.g. automated checks, compliance scans).