Lead Observability Engineer

Overview

Remote
On Site
Full Time
Part Time
Accepts corp to corp applications
Contract - Independent
Contract - W2

Skills

Observability
Sumo logic
eks
SRE
AWS
Dynatrace
migration

Job Details

JD:

Experience: 10+ years (with 3+ years in Sumo Logic & Cloud-native observability)

Job Summary:

We are seeking a highly skilled Lead Observability Engineer to lead a critical implementation of Sumo Logic for a client migrating from Dynatrace. This role requires deep expertise in Sumo Logic, Site Reliability Engineering (SRE) practices, and Kubernetes (EKS) observability. The ideal candidate will design and implement scalable dashboards, alerts, and tracing strategies, drive service-level reliability, and enable a steady-state SRE operations model.

Key Responsibilities:

Lead the end-to-end implementation of Sumo Logic observability platform for AWS and EKS environments.

Migrate monitoring and alerting assets from Dynatrace to Sumo Logic.

Define and implement SLIs/SLOs, error budgets, and reliability metrics for containerized services.

Deploy and configure Sumo Logic collectors across AWS and Kubernetes workloads (EKS).

Configure log, metric, and trace ingestion pipelines using OpenTelemetry and Sumo Logic apps.

Design and maintain dashboards for service health, performance, and reliability insights.

Implement intelligent alerting and notification workflows, using thresholds, baselines, and anomaly detection.

Collaborate with DevOps, SRE, and development teams to ensure complete tracing coverage across services.

Ensure best practices for alert noise reduction, escalation policies, and incident response are in place.

Contribute to observability runbooks, operational handover, and training for the client SRE team.

Required Skills & Qualifications:

Expert-level experience with Sumo Logic, including dashboarding, alerting, collector deployment, and ML features.

Strong background in Site Reliability Engineering (SRE), including SLIs/SLOs, error budgets, MTTR/MTTD metrics.

Proficiency in AWS services (especially CloudWatch, CloudTrail, Lambda, RDS) and EKS (Amazon Kubernetes Service).

Hands-on experience with OpenTelemetry for distributed tracing and service maps.

Strong understanding of Kubernetes metrics, pod health, container resource usage, and cluster monitoring.

Proven ability to define alert thresholds, configure notification routing (e.g. Slack, PagerDuty, ServiceNow), and manage alert fatigue.

Strong scripting experience with tools like Terraform, Helm, YAML, and GitOps workflows.

Experience with incident triage, RCA documentation, and building operational maturity in observability teams.

Excellent communication and stakeholder engagement skills.

Preferred Qualifications:

Sumo Logic certifications (Admin, Advanced Analytics) are a plus.

Experience with Dynatrace (for migration purposes).

Familiarity with integrating observability into CI/CD pipelines.

Exposure to service mesh (Istio/Linkerd) and monitoring microservices in that context.

Deliverables This Role Will Drive:

Sumo Logic observability reference architecture

EKS and AWS observability configuration

SLI/SLO documentation and tracking

Alerting and tracing setup across services

Production-ready dashboards and runbooks

Knowledge transfer and enablement sessions for SRE/DevOps teams

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.