Senior Data Reliability Engineer (CI/CD & Data Platform)

Data Reliability Engineer (CI/CD & Data Platform)

Job Summary

We are looking for a Data Reliability Engineer to ensure the reliability, scalability, and operational excellence of our data platform. The role focuses on building resilient data pipelines, implementing CI/CD for data engineering workflows, automating deployments, monitoring production systems, and maintaining high standards of data quality and availability.

Key Responsibilities

• Design, build, and maintain reliable batch and streaming data pipelines.
• Develop and manage CI/CD pipelines for data applications, ETL workflows, and infrastructure.
• Automate build, test, deployment, and rollback processes for data solutions.
• Monitor data pipelines, platforms, and infrastructure using observability tools.
• Troubleshoot production incidents and perform root cause analysis (RCA).
• Implement data quality validation, testing, and monitoring.
• Manage Infrastructure as Code (IaC) using Terraform or CloudFormation.
• Collaborate with Data Engineering, DevOps, Platform Engineering, QA, and Analytics teams.
• Optimize Spark jobs, SQL workloads, and cloud data services for performance and reliability.
• Ensure high availability, disaster recovery, backup, and security compliance.
• Define and monitor SLAs, SLOs, and reliability metrics.
• Support release management, version control, and environment promotion strategies.

Required Technical Skills

Programming

• Python
• SQL
• Shell Scripting
• Java or Scala (preferred)

Data Engineering

• Apache Spark
• Apache Airflow
• dbt
• Kafka
• Snowflake, Databricks, Redshift, BigQuery, or Synapse

CI/CD

• Jenkins
• GitHub Actions
• GitLab CI/CD
• Azure DevOps Pipelines
• Bitbucket Pipelines

Version Control

• Git
• GitHub
• GitLab
• Bitbucket

Containers & Orchestration

• Docker
• Kubernetes

Cloud Platforms

• AWS
• Azure
• Google Cloud Platform (Google Cloud Platform)

Infrastructure as Code

• Terraform
• CloudFormation
• Azure Bicep (preferred)

Monitoring & Observability

• Grafana
• Prometheus
• Datadog
• Splunk
• ELK Stack
• CloudWatch / Azure Monitor / Google Cloud Monitoring

Testing

• Unit Testing
• Integration Testing
• Data Quality Testing
• Great Expectations
• Soda
• PyTest

CI/CD Responsibilities

• Build automated pipelines for data applications and ETL deployments.
• Integrate automated testing into deployment pipelines.
• Implement deployment strategies such as blue-green and rolling deployments where appropriate.
• Automate schema migrations and database deployment processes.
• Enforce code quality through static code analysis and pull request validation.
• Implement secure secret management and credential rotation.
• Maintain release pipelines across development, QA, staging, and production environments.
• Monitor deployment success rates and automate rollback mechanisms.