Observability Developer

Senior Observability Developer (Dynatrace & BigPanda)

Location: McLean , VA

Duration : long term 

 

The Role

We are seeking a highly skilled and passionate Observability Developer to join our Platform Engineering team. You will be the cornerstone of our observability initiative, responsible for designing, building, and maintaining the platforms and integrations that give our development and SRE teams the power to understand system behavior, troubleshoot issues proactively, and ensure optimal performance.

 

Your primary focus will be on maximizing the value of our core tools, Dynatrace and BigPanda, by creating custom integrations, automating workflows, and developing a data-driven culture of observability.

 

Key Responsibilities

Platform Development & Integration:

·  Design, develop, and maintain robust integrations between Dynatrace, BigPanda, and other systems in our tech stack (e.g., CI/CD pipelines, ITSM tools like ServiceNow, data warehouses).

·  Develop custom extensions, plugins, and ActiveGates for Dynatrace to monitor technologies not covered out-of-the-box.

·  Build and optimize BigPanda alert correlation rules and workflows to reduce noise and provide actionable, context-rich incidents.

·  Create custom dashboards, reports, and data visualizations in Dynatrace to provide tailored views for different engineering teams.

Automation & Orchestration:

·  Automate the deployment and configuration of observability agents and monitoring as code using tools like Terraform, Ansible, or Python.

·  Develop automated remediation scripts triggered by Dynatrace events or BigPanda incidents.

·  Implement self-healing systems to automatically resolve common operational issues.

Data Analysis & Insight:

·  Leverage the Dynatrace API and DQL (Dynatrace Query Language) to extract, analyze, and correlate performance, log, and business data.

·  Work with BigPanda’s API to enrich alert data and automate incident response processes.

·  Identify trends and patterns in system behavior to predict and prevent potential outages.

 

Required Qualifications & Skills

·  3+ years of hands-on experience in an Observability, SRE, or DevOps role with a strong development background.

·  Proven Expertise in Dynatrace: Deep, hands-on experience with the Dynatrace platform, including:

o    Application and Infrastructure Monitoring

 

o    Davis AI and problem analysis

 

o    DQL (Dynatrace Query Language) for advanced data analysis

 

o    Custom metric ingestion and API usage

 

o    Setting up Synthetic Monitoring

 

o    Experience with Dynatrace ActiveGates and OneAgent deployment/configuration.

 

·  Proven Expertise in BigPanda: Significant experience using BigPanda for event correlation and alert management, including:

 

o    Building and tuning correlation rules

 

o    Incident enrichment and workflow automation

 

o    Integration with third-party systems

 

·  Strong Programming/Scripting Skills: Proficiency in at least one language such as Python, Go, Java, or JavaScript/Node.js for automation and integration development.

 

·  Infrastructure as Code (IaC): Experience with Terraform, Ansible, or similar tools for managing configuration.

 

·  API Expertise: Strong understanding of RESTful APIs and experience building integrations.

 

·  Solid understanding of cloud platforms (AWS, Azure, or Google Cloud Platform) and container technologies (Docker, Kubernetes).

 

Preferred Qualifications

·  Dynatrace Certified Associate/Professional or similar certification.

·  Experience with other observability tools (e.g., Prometheus, Grafaa, Splunk, Datadog).

·  Knowledge of OpenTelemetry (OTel) for instrumentati

·  Experience in an SRE role, practicing error budget management and SLI/SLO definition.

·  Familiarity with AIOps concepts and practical implementation.