Senior Monitoring and Observability Engineer

This role is responsible for monitoring coverage, platform integration, agent deployment, tagging and normalization, dashboards, alerting, logs, APM, synthetic monitoring, automation, and operational integrations. Datadog is the primary enterprise observability platform used in the environment. Strong Datadog experience is preferred, however, candidates with substantial experience engineering and operating other enterprise monitoring or observability platforms will be considered where they demonstrate strong transferable monitoring expertise and the ability to rapidly develop proficiency with new technologies.

The engineer partners with Operations and engineering teams to improve visibility, alert quality, incident detection, troubleshooting, performance analysis, and operational reliability across the enterprise.

Primary Responsibilities

In this Role you will:

• Engineer, operate, maintain, and continuously improve the enterprise monitoring and observability platform, including dashboards, monitors, metrics, logs, APM, synthetic monitoring, tagging, integrations, and related capabilities.
• Assess monitoring coverage across enterprise systems and applications, identify visibility gaps, and coordinate onboarding or remediation with the appropriate technical teams.
• Maintain monitoring coverage across Windows, Linux, cloud, OpenShift/Kubernetes, virtualized, database, network, storage, middleware, and application environments.
• Support monitoring and observability for Red Hat OpenShift, Kubernetes, OpenShift Virtualization, and virtual machine workloads running on OpenShift.
• Configure and troubleshoot monitoring agents, integrations, collectors, APIs, and related platform components.
• Build and maintain consistent tagging, metadata, dashboards, alerts, service health views, and operational reporting.
• Automate monitoring deployment, configuration, tagging, onboarding, upgrades, and integrations using Ansible, APIs, scripting, CI/CD, infrastructure-as-code, or similar technologies.
• Develop and maintain integrations between observability platforms, ServiceNow, notification systems, on-call workflows, and other enterprise operational systems.
• Use monitoring and observability data to troubleshoot complex performance and availability issues, support incident response and root-cause analysis, and recommend technical remediation.
• Correlate infrastructure, application, platform, and dependency telemetry to identify service degradation and recurring technical issues.
• Partner with Operations and engineering teams to improve monitoring coverage, alert quality, service visibility, incident detection, escalation, and operational response.
• Analyze telemetry and historical trends to identify capacity risks, recurring issues, monitoring gaps, and opportunities for improvement.
• Develop actionable performance, availability, capacity, and monitoring coverage reporting for technical and leadership stakeholders.
• Maintain monitoring standards, technical documentation, configuration guidance, and operational procedures.

Basic Qualifications

• BS degree and 8-12 years of prior relevant experience, or Master's degree with 6-10 years of prior relevant experience. Additional relevant experience may be considered in lieu of degree requirements where permitted by contract.
• Strong hands-on experience engineering and operating enterprise monitoring or observability platforms.
• Strong Datadog experience is preferred; however, substantial experience with ScienceLogic SL1, SolarWinds, Dynatrace, New Relic, Splunk Observability, LogicMonitor, PrometheGrafana, or comparable enterprise platforms will be considered based on demonstrated monitoring and observability engineering expertise.
• Demonstrated ability to apply monitoring and observability engineering principles across technologies and rapidly develop proficiency with new platforms.
• Production experience monitoring Windows and Linux infrastructure and Kubernetes or Red Hat OpenShift environments.
• Experience deploying, configuring, upgrading, and troubleshooting monitoring agents, integrations, dashboards, alerts, tagging, and operational reporting.
• Experience automating monitoring deployment or administration using Ansible, APIs, scripting, CI/CD pipelines, infrastructure-as-code, or similar technologies.
• Experience integrating monitoring or observability platforms with ITSM systems such as ServiceNow.
• Strong troubleshooting and dependency-analysis skills across infrastructure, applications, networks, platforms, and services.
• Ability to analyze technical telemetry, identify monitoring or performance gaps, and translate findings into actionable recommendations.
• Ability to communicate technical findings and recommendations to technical teams, project leadership, and customer stakeholders.

Preferred Qualifications

• Direct experience engineering or administering Datadog in a large enterprise environment.
• Experience with application performance monitoring, distributed tracing, or OpenTelemetry.
• Experience with Datadog APM, Log Management, Synthetic Monitoring, RUM, Network Performance Monitoring, Database Monitoring, or related capabilities.
• Experience with Red Hat OpenShift Virtualization, CNV, KubeVirt, or related Kubernetes-based virtualization technologies.
• Experience monitoring Microsoft Azure or AWS environments.
• Experience with Terraform, monitoring-as-code, API-driven deployment, or related infrastructure-as-code approaches.
• Experience supporting federal agency IT environments governed by FISMA, FedRAMP, NIST, or related security requirements.
• Relevant technical certifications such as Datadog, AWS, Microsoft Azure, Red Hat OpenShift, Terraform, or ITIL are preferred.