HPC consultant

Role- HPC consultant

Location :  Remote 
 
 HPC Cluster & Scheduler Management

• Design, configure, tune, and optimize SLURM partitions, queues, QoS, and scheduling policies to maximize cluster utilization and workload efficiency.

• Perform in-depth analysis of job scheduling behavior, bottlenecks, and resource contention.

• Troubleshoot job failures, performance degradation, and scheduler-related issues in production HPC environments.

• Implement fair-share, backfill, reservations, and policy-driven scheduling as required.

Storage Benchmarking & Procurement Support

• Lead HPC storage performance benchmarking using industry-standard tools (e.g., IOR, FIO, MDTest, IOzone).

• Analyze I/O patterns of HPC workloads and map them to appropriate storage architectures (parallel file systems, NVMe, Lustre, Spectrum Scale, etc.).

• Provide technical input for storage selection and procurement, including performance expectations, sizing, and cost-performance tradeoffs.

• Collaborate with vendors and internal teams during POCs and performance validation exercises.

HPC Application Build & Optimization

• Build, install, configure, and maintain HPC applications, compilers, libraries, and scientific software stacks.

• Optimize application performance using MPI, OpenMP, GPU acceleration (where applicable), and tuned math libraries.

• Support multiple compiler toolchains (GCC, Intel, LLVM, NVIDIA HPC SDK, etc.).

• Implement and manage environment modules (Lmod) or similar software management frameworks.

System Performance & Operations

• Conduct system-level performance tuning across compute, memory, network, and storage layers.

• Diagnose node-level issues involving CPU, GPU, interconnects (InfiniBand/Ethernet), and OS configurations.

• Create operational runbooks, performance baselines, and troubleshooting documentation.

• Support cluster upgrades, expansions, and hardware refresh activities.

Collaboration & Delivery

• Work closely with application owners, researchers, and infrastructure teams to meet aggressive delivery timelines.

• Translate workload requirements into practical HPC configurations and optimizations.

• Provide clear technical guidance and recommendations to leadership and stakeholders.

Required Skills & Experience 
Core HPC Skills

• 8 12+ years of hands-on HPC engineering experience in production environments.

• Strong expertise with SLURM (configuration, tuning, troubleshooting).

• Solid understanding of Linux systems (RHEL/CentOS/Rocky/Alma preferred).

• Deep knowledge of HPC storage systems and I/O performance analysis.

• Proven experience building and optimizing HPC applications and libraries.

Technical Proficiency

• MPI implementations (Open MPI, MPICH), OpenMP

• Compilers and toolchains (GCC, Intel, NVIDIA HPC SDK)

• Performance tools (perf, vtune, nvprof/nsys, IB diagnostics)

• Environment modules (Lmod), package managers (Spack preferred)

• Bash/Python scripting for automation and diagnostics

  
Nice to Have

• Experience with GPU-based HPC workloads (NVIDIA CUDA, ROCm).

• Exposure to cloud-based HPC (Azure, AWS, Google Cloud Platform).

• Familiarity with parallel file systems such as Lustre or IBM Spectrum Scale.

• Vendor engagement experience for HPC hardware/storage evaluations.