Job Title: Staff Observability Platform Engineer (AI / GPU Infrastructure)
Location: Seattle, WA / Houston, TX / New York, NY (Onsite)
Employment Type: Contract
Position Overview
We are looking for a highly experienced Staff Observability Platform Engineer to design, build, and operate large-scale observability platforms supporting AI/GPU infrastructure and Kubernetes environments.
This is not a traditional monitoring or dashboard role. You''ll own the observability backend platform, ensuring it remains scalable, reliable, and cost-efficient as telemetry volumes grow.
Key Responsibilities
- Design, build, and scale enterprise metrics, logging, tracing, and telemetry platforms.
- Architect and operate distributed observability backends using Prometheus, Mimir, Thanos, VictoriaMetrics, Cortex, Loki, Elasticsearch, or similar technologies.
- Build and optimize OpenTelemetry Collector pipelines including routing, filtering, sampling, and exporters.
- Optimize cardinality, ingestion, retention, storage, query performance, and infrastructure cost.
- Manage large-scale Kubernetes observability across multi-cluster environments.
- Troubleshoot production issues involving metrics, logs, traces, networking, storage, and distributed systems.
- Write and review production-quality code and establish observability standards across engineering teams.
- Partner closely with Platform, Infrastructure, Security, and Application Engineering teams.
Required Qualifications
- 8+ years of experience in Observability, Platform Engineering, SRE, or Infrastructure Engineering.
- Hands-on experience operating production-scale observability platforms such as Mimir, Thanos, VictoriaMetrics, Cortex, Loki, or Elasticsearch.
- Strong expertise with Prometheus, OpenTelemetry, and telemetry pipeline design.
- Experience managing large-scale production environments with measurable metrics (ingestion rates, active time series, storage, retention, cluster size, etc.).
- Strong understanding of cardinality, retention strategies, storage architecture, sampling, query optimization, and cost management.
- Deep hands-on experience with Kubernetes, distributed systems, networking, service discovery, autoscaling, and reliability engineering.
- Strong programming skills in Go and/orn Python.
- Solid understanding of production engineering concepts including retries, backpressure, buffering, circuit breaking, graceful degradation, scalability, and failure handling.
Preferred Qualifications
- Experience with GPU infrastructure, AI/ML platforms, or HPC environments.
- Knowledge of NVIDIA DCGM, InfiniBand, RoCE/RDMA, NVLink, NVSwitch, NCCL, or Slurm.
- Experience building custom Prometheus Exporters, OpenTelemetry Collectors, or large multi-cluster Kubernetes observability platforms.
Ideal Candidate
We''re looking for someone who has personally owned and operated observability backends at production scale, understands the trade-offs between cardinality, retention, storage, performance, and cost, and can build scalable observability platforms for modern AI/GPU infrastructure. Experience limited to dashboards, alerts, or consuming monitoring tools without backend ownership will not be sufficient for this role.
Never miss an opportunity! Create an alert based on the job you applied for.