Role: Production Engineer - HPC Reliability & Automation
Location: United States
Duration: Fulltime (No Contractors)
Work Type: Remote-first, with occasional on-site visits to customer offices. Candidates located near any Meta office location will be preferred.
Tech Mahindra is seeking a highly skilled Production Engineer - HPC Reliability & Automation to support and enhance large-scale cloud and High-Performance Computing (HPC) environments powering advanced AI, analytics, and accelerated computing workloads.
This role focuses on reliability engineering, operational excellence, automation, observability, and performance optimization across distributed compute infrastructure. The ideal candidate will have deep expertise in Linux systems, large-scale distributed environments, troubleshooting complex production issues, and building intelligent automation solutions that improve operational efficiency and platform stability.
Success in this role requires strong problem-solving capabilities, operational ownership, and the ability to quickly understand complex infrastructure ecosystems spanning compute, scheduling, storage, networking, cloud platforms, and accelerator technologies.
Key Responsibilities
- Develop deep technical understanding of large-scale HPC and cloud infrastructure environments, including compute clusters, workload schedulers, accelerator platforms, storage systems, networking, and authentication services.
- Proactively identify reliability, scalability, and performance issues by correlating logs, metrics, telemetry, and events across multiple cloud and on-premises environments.
- Troubleshoot complex production issues across Linux systems, containers, distributed applications, schedulers, storage, networking, and compute infrastructure.
- Design and build automation solutions that improve detection, diagnosis, remediation, and operational efficiency while ensuring safety, auditability, and reliability.
- Enhance observability, monitoring, alerting, and operational intelligence capabilities to reduce downtime and improve platform health.
- Collaborate with infrastructure, cloud, security, and engineering teams to implement long-term solutions and improve service resiliency.
- Participate in incident management, root cause analysis, postmortems, and continuous improvement initiatives.
- Develop tooling, scripts, and platform automation to minimize manual operational activities.
Required Skills & Experience
- Strong experience supporting production HPC environments, large-scale distributed compute platforms, GPU-accelerated infrastructure, or reliability-focused engineering environments.
- Deep Linux administration, troubleshooting, and performance engineering experience.
- Strong understanding of networking fundamentals, DNS, authentication, storage I/O, containers, and distributed system operations.
- Hands-on experience with observability and monitoring platforms such as:
- Prometheus
- Grafana
- OpenTelemetry
- ELK Stack
- Datadog
- Splunk
- Strong programming and automation experience using:
- Python
- Go (preferred)
- Shell/Bash scripting
- Experience conducting complex production investigations from issue identification through root cause analysis, resolution, validation, and long-term prevention.
- Experience supporting highly available production platforms with operational ownership responsibilities.
Preferred Qualifications
- Experience with workload orchestration and scheduling platforms such as:
- Kubernetes
- Slurm
- OpenShift
- Similar cluster management solutions
- Experience supporting AI, machine learning, data science, or GPU-intensive computing environments.
- Understanding of distributed training infrastructure, checkpoint management, job scheduling, workload placement, and compute cluster operations.
- Experience working with cloud platforms such as:
- AWS
- Microsoft Azure
- Google Cloud Platform (Google Cloud Platform)
- Similar large-scale cloud providers
- Experience with enterprise-scale storage technologies, including:
- NFS
- POSIX-compliant storage
- Lustre
- Object Storage
- High-performance storage systems
- Experience with Infrastructure as Code and platform automation tools:
- Terraform
- Helm
- Kubernetes Operators
- Ansible
- Experience designing and validating resilient infrastructure through testing, fault simulation, workload replay, and controlled recovery procedures.
- Exposure to AI-assisted operations, intelligent automation, or autonomous remediation workflows is beneficial.
