Senior Observability Engineer

Job Title: Sr. Observability Engineer
Location: Headquarters / Telecommute
Classification (HR only): Exempt Non-Exempt
Reports To (Title): COO Widescope Consulting and Contracting

JOB SUMMARY

We are looking for a Senior Level Observability Engineer with proven past performance in an Observability Engineering capacity and a high-level proficiency in utilizing Dynatrace and/or ELK. The Candidate has strong test-worthy familiarity with writing scripts in languages like Python and key experience automating tasks. The selected candidate will be joining a team of skilled engineers with a broad background in enterprise monitoring and Observability. As an Observability Engineer, you will be focused on maintaining the reliability, scalability and availability of our Log management solution as well as our Metrics and Observability platform which heavily uses automation (terraform, Ansible and scripts), this role requires maintaining performance KPI of our solutions and defining their SLOs.

Role Capabilities:

• Maintain and deploy monitoring and alerting.
• Design, configuration and maintenance of log aggregation solution at a large scale.
• Set up and manage ingestion pipelines and data transformations
• Have the mindset of automate any task .
• Monitoring and Alerting: Build and maintain robust monitoring systems using tools like Elk, Dynatrace, Prometheus and Grafana to detect potential issues early and trigger alerts for timely response.
• Maintain associated documentation as it applies to our audit and certification requirements
• Participate in troubleshooting, capacity planning, and performance analysis activities
• Research new monitoring requirements and in many cases write code to accomplish.
• Strong expertise in setting up monitoring policies/rules/templates; and writing scripts to accomplish monitoring requirements.

JOB QUALIFICATIONS

Required:

• BS/MS in CS/engineering or equivalent, OR 5+ years of experience.
• 3+ years of experience working directly with monitoring tools as either an Admin, SME or as an Architect, preferably with Dynatrace and/or ELK.
• Hands-on experience with designing data pipelines using Logstash and/or fluentbit/fluentd.
• Fluent in writing scripts in languages like Python and (Bash or powershell) to automate tasks.
• Experience in Terraform and Ansible. Syntax, best practices, and managing complex configurations in multi commercial and Gov clouds to build and manage infra and applications.
• Very good working knowledge with Linux OS.
• Highly self-motivated and directed
• Good analytical and problem-solving/troubleshooting abilities.

Preferred:

• Knowledge of SNMP, TCP dump and tracing.
• Knowledge of AIOPS platform.
• Other scripting experience (JavaScript, Java, PowerShell, or others)