ABOUT THE ROLE
We''re hiring a Principal/Staff Observability Platform Engineer to own the technical direction of our observability platform — the systems that give us deep visibility into GPU clusters, AI workloads, and the infrastructure running them.
This is a "define, build, and lead" role, not a "maintain and operate" role. You''ll set the architectural roadmap, raise the engineering bar across teams, and make sure the platform scales ahead of the business, not behind it. We have a strong bias toward simplicity — the systems you build should be easy to operate and self-evidently correct when something goes wrong.
RESPONSIBILITIES
- Own the technical strategy and architecture for observability across metrics, logs, traces, and alerting at scale
- Drive platform decisions with multi-year impact: tooling, data models, ingestion patterns, retention, cardinality management
- Identify systemic gaps before they become incidents; design platforms that make failure visible and fast to diagnose
- Partner with SRE, infrastructure, and AI/ML teams to embed observability natively into how we build and operate
- Define standards and patterns that other engineers adopt because they''re clearly better, not by mandate
- Mentor and technically grow the observability team
- Lead incident postmortems and drive durable platform improvements
- Evaluate and introduce tooling that improves signal quality, operational efficiency, or scalability — and retire what doesn''t
REQUIRED SKILLS & EXPERIENCE
- 8+ years in SRE, infrastructure engineering, platform engineering, or observability-focused roles
- Proven experience operating observability infrastructure at serious scale
- Deep hands-on experience with a significant subset of: Prometheus, Thanos, VictoriaMetrics, Grafana, Loki, Tempo, OpenTelemetry, ClickHouse, Elastic
- Strong engineering fundamentals in Python, Go, or similar
- Kubernetes at scale
- Infrastructure-as-Code as default practice (Terraform, Ansible, or equivalent)
- Demonstrated ability to architect systems, write code, review others'' work, and clearly explain tradeoffs
- Track record of influencing engineering direction across teams without formal authority
PREFERRED SKILLS & EXPERIENCE
- Experience with high-volume streaming pipelines for observability data (Kafka, Vector, Fluent Bit, etc.)
- Background in AI/ML infrastructure observability: GPU utilization, training job visibility, inference latency
- Familiarity with GPU infrastructure or HPC environments (Slurm)
- Prior experience defining observability strategy at an organizational level
EQUAL OPPORTUNITY
We strongly encourage applications from people of color, the LGBTQ+ community, people with disabilities, neurodivergent individuals, parents, carers, and people from lower socio-economic backgrounds. If there''s anything we can do to accommodate your specific situation, please let us know.
Never miss an opportunity! Create an alert based on the job you applied for.
