Observability Engineer (Google Cloud Platform & OpenTelemetry Specialist)

Observability Engineer (Google Cloud Platform & OpenTelemetry Specialist)
Remote
12+ months We are seeking an expert in Cloud Observability to lead the design, implementation, and optimization of our monitoring infrastructure using OpenTelemetry (OTel) Collectors. You will be responsible for bridging the gap between our Google Cloud Platform infrastructure and our monitoring backends, ensuring that our telemetry data is high-quality, cost-effective, and actionable. Core Responsibilities
OTel Collector Architecture: Design and deploy scalable OTel Collector pipelines (as sidecars, DaemonSets, or standalone services) on GKE and Compute Engine.
Infrastructure Monitoring: Configure OTel receivers (e.g., hostmetrics, kubeletstats) to capture critical Google Cloud Platform resource health data.
Label Enrichment & Context: Implement the resourcedetection processor to automatically inject Google Cloud Platform-specific metadata (Project ID, Zone, Instance ID, etc.) into all telemetry signals.
Pipeline Optimization: Utilize batch, memory_limiter, and transform processors to manage data volume, prevent OOM issues, and reduce Google Cloud Platform Cloud Monitoring costs.
Backend Integration: Configure the googlecloud exporter to seamlessly route metrics to Google Cloud Monitoring and Managed Service for Prometheus.
Required Technical Expertise 1. Google Cloud Platform-Specific Monitoring Strategy

• The ideal candidate knows that "collecting everything" is a recipe for high costs and low signal. You should have deep experience with:
• High-Value Metrics: Identifying and prioritizing "Golden Signals" (Latency, Traffic, Errors, Saturation) across Google Cloud Platform services like Cloud Run, GKE, Pub/Sub, and Cloud SQL.
• Quota & Limit Management: Understanding Google Cloud Platform's metric descriptor limits (10,000 per project) and avoiding "Cardinality Explosions" from OTel labels.
• IAM & Security: Setting up the necessary roles/monitoring.metricWriter and roles/compute.viewerpermissions for the Collector.

2. Advanced OTel Configuration

• Label Enrichment: Expert-level knowledge of using the Resource Detection Processor with the Google Cloud Platform detector to ensure metrics are correctly mapped to Google Cloud Platform Monitored Resources.
• Processor Logic: Proficiency in the Transform Processor (OTTL) to rename metrics or drop unnecessary labels before they reach the backend.
• Deployment Patterns: Experience managing OTel configurations via Helm, Terraform, or the OTel Operator for Kubernetes.

Preferred Qualifications

• Experience with Google Managed Service for Prometheus (GMP).
• Strong understanding of the OTLP (OpenTelemetry Protocol) and how it maps to Google Cloud Platform's CUMULATIVE and GAUGEmetric types.
• Ability to build dashboards in Cloud Monitoring that correlate infrastructure metrics with application-level traces.