Position: Network and Compute Engineer, AI Infrastructure
Location: San Jose, CA / REMOTE WITH TRAVEL
CONTRACT
Job Description:
Position Summary
Network and Compute Engineer to design, deploy, validate, and operate high-performance infrastructure supporting enterprise artificial intelligence and accelerated-computing workloads.
The engineer will work across compute, networking, operating systems, container orchestration, cluster management, storage integration, security, and observability. The successful candidate will be comfortable working hands-on with complex infrastructure, diagnosing issues across multiple technical layers, and translating platform requirements into reliable and repeatable operational solutions.
The role is aligned to an NVIDIA AI Factory environment. The source material specifically covers NVIDIA Mission Control, Base Command Manager, Kubernetes installation, rack management, bare-metal management, observability, Run:ai, MGX architecture, Slurm, GB200 software and firmware, NVLink partitioning, XID troubleshooting, and telemetry.
Key Responsibilities
Compute and Cluster Engineering
- Deploy, configure, validate, and operate accelerated-compute clusters.
- Perform server and cluster bring-up, including operating-system imaging, boot validation, firmware alignment, hardware discovery, and post-deployment health checks.
- Support NVIDIA MGX-based infrastructure and understand the relationship between hardware architecture, firmware, operating systems, drivers, and the platform software stack.
- Install and administer Linux-based compute environments.
- Support Kubernetes installation, configuration, upgrades, and lifecycle operations.
- Deploy and administer Slurm-based workload environments.
- Assist with NVIDIA GB200 firmware updates, software-stack validation, sanity checks, and platform readiness testing.
- Support NVLink configuration and partitioning, where applicable.
- Troubleshoot GPU and system errors, including XID events, degraded nodes, boot failures, firmware inconsistencies, and workload-performance issues.
- Develop standardized, repeatable build and validation procedures for compute nodes and clusters.
Network Engineering
- Design and support resilient network connectivity for accelerated-compute and AI cluster environments.
- Configure and troubleshoot high-speed Ethernet and, where applicable, InfiniBand fabrics.
- Support leaf-spine network architectures, network segmentation, routing, switching, addressing, name resolution, and time synchronization.
- Diagnose packet loss, latency, congestion, interface errors, cabling faults, routing issues, and workload connectivity failures.
- Validate host-to-host, node-to-storage, management-plane, and workload-plane connectivity.
- Support network configuration required for Kubernetes, cluster provisioning, telemetry, and distributed workloads.
- Work with security teams to implement network isolation, firewall rules, access controls, and secure administrative paths.
- Maintain accurate network diagrams, port maps, addressing plans, and configuration records.
Platform Provisioning and Management
- Install, configure, and operate NVIDIA Base Command Manager.
- Support rack import, rack view, base view, node provisioning, and cluster lifecycle activities.
- Work with bare-metal management systems for server discovery, power operations, hardware health, firmware administration, and remote troubleshooting.
- Integrate cluster platforms with enterprise directory services.
- Support NVIDIA Mission Control processes and related AI Factory operational practices.
- Assist with the deployment and operation of Run:ai or equivalent workload-orchestration capabilities.
- Automate repetitive provisioning, validation, configuration, and reporting activities.
- Implement infrastructure monitoring for compute nodes, GPUs, network devices, interfaces, firmware, thermals, power, storage dependencies, and cluster services.
- Establish actionable dashboards, telemetry, alerts, and operational health checks.
- Lead technical diagnosis across hardware, firmware, Linux, Kubernetes, scheduler, network, and application layers.
- Perform root-cause analysis and document corrective and preventive actions.
- Develop operational runbooks for failure scenarios such as node loss, degraded network links, GPU errors, provisioning failures, and cluster-service disruption.
- Participate in incident response, problem management, capacity planning, lifecycle planning, and controlled change implementation.
- Identify automation and standardization opportunities that reduce manual effort, deployment defects, and recovery time.
- Work with infrastructure, cloud, security, storage, application, vendor, and service-management teams.
Security and Compliance Responsibilities
- Apply least-privilege access to management, compute, network, and orchestration platforms.
- Integrate administrative access with approved enterprise identity and directory services.
- Secure management interfaces, service accounts, automation credentials, application programming interfaces, and remote-access pathways.
- Maintain separation between management, workload, storage, and out-of-band management networks.
- Support firmware, operating-system, container, Kubernetes, and network vulnerability remediation.
- Ensure administrative actions and infrastructure changes are traceable through logging and change records.
- Protect sensitive cluster configuration, credentials, certificates, logs, and telemetry.
- Follow secure configuration standards, change controls, and applicable customer or regulatory requirements.
Required Technical Skills
- Strong administration and troubleshooting experience with Linux.
- Strong knowledge of data-center networking, including Ethernet, Transmission Control Protocol/Internet Protocol, routing, switching, virtual local area networks, link aggregation, Domain Name System, Dynamic Host Configuration Protocol, and Network Time Protocol.
- Hands-on experience installing or operating Kubernetes clusters.
- Experience deploying and supporting physical compute infrastructure.
- Understanding of server boot processes, operating-system imaging, firmware, device drivers, hardware management, and remote management controllers.
- Experience diagnosing issues across server, operating-system, network, and application layers.
- Working knowledge of Python, Bash, Ansible, or equivalent infrastructure-automation technologies.
- Experience with monitoring, logging, metrics, dashboards, alerting, and telemetry.
- Ability to create clear architecture diagrams, build documents, operating procedures, and troubleshooting runbooks.
- Strong analytical skills and the ability to work effectively during complex infrastructure incidents.
Preferred Technical Skills
- Experience with NVIDIA AI infrastructure, NVIDIA Mission Control, or NVIDIA Base Command Manager.
- Experience with NVIDIA MGX or GB200-based systems.
- Experience with Slurm and distributed-compute scheduling.
- Experience with Run:ai.
- Experience with InfiniBand, Remote Direct Memory Access, or Remote Direct Memory Access over Converged Ethernet.
- Knowledge of NVLink and multi-GPU communication.
- Experience interpreting NVIDIA XID errors and GPU telemetry.
- Experience with bare-metal provisioning and Redfish-compatible management interfaces.
- Experience with Prometheus, Grafana, OpenTelemetry, or equivalent observability platforms.
- Exposure to container runtimes, container registries, Kubernetes networking, and graphics-processing-unit operators.
- Experience supporting production AI, machine-learning, high-performance-computing, or large-scale data platforms.
Qualifications
- Bachelor’s degree in computer science, information technology, electronics, telecommunications, or a related field, or equivalent practical experience.
- Recommended 7 or more years of experience in infrastructure, compute, network, data-center, cloud-platform, or high-performance-computing engineering.
- Recommended 3 or more years of hands-on experience with Linux-based cluster platforms, Kubernetes, accelerated computing, or comparable large-scale infrastructure.
- Demonstrated experience owning complex technical problems from initial diagnosis through resolution and prevention.
Relevant Certifications
Certifications are desirable but should not replace proven hands-on capability:
- NVIDIA Certified Associate or NVIDIA Certified Professional credentials relevant to AI infrastructure
- Red Hat Certified System Administrator or Red Hat Certified Engineer
- Certified Kubernetes Administrator
- Cisco Certified Network Professional, Data Center or Enterprise
- NVIDIA networking or InfiniBand training
- Linux Foundation certifications
- Security-focused credentials appropriate to the operating environment
