Lead Site Reliability Engineer (Splunk)

Job Title: Lead Site Reliability Engineer (SRE)
Location: Remote
Job Type: Contract

Job Overview

We are seeking an experienced Lead Site Reliability Engineer (SRE) to lead initiatives focused on improving the reliability, scalability, performance, and operational efficiency of enterprise observability platforms.

The ideal candidate will have strong hands-on experience with Splunk Enterprise or Splunk Cloud administration, Elasticsearch, Prometheus, Grafana, Grafana Tempo, OpenTelemetry, and distributed tracing. This role requires expertise in building and maintaining large-scale monitoring and observability infrastructure, developing dashboards and alerts, and automating infrastructure using Terraform and Infrastructure as Code (IaC).

Key Responsibilities

• Design, deploy, administer, and maintain enterprise-scale observability platforms.
• Build and support Splunk Enterprise or Splunk Cloud infrastructure, including Indexers, Search Head Clusters, Heavy Forwarders, and Deployment Servers.
• Develop and optimize Splunk Search Processing Language (SPL) queries, dashboards, alerts, and operational analytics.
• Deploy, manage, and troubleshoot large-scale Elasticsearch clusters for log analytics and search.
• Design and maintain distributed tracing platforms using Grafana Tempo and OpenTelemetry.
• Build end-to-end tracing pipelines, establish instrumentation standards, and implement trace retention strategies.
• Scale and optimize Prometheus, Grafana, Kafka, Tempo, and OpenTelemetry-based monitoring solutions.
• Develop dashboards, alerts, and visualizations using Splunk, Grafana, Kibana, and Grafana Tempo.
• Implement comprehensive observability solutions covering metrics, logs, and distributed traces.
• Automate infrastructure provisioning, configuration, and operational tasks using Terraform and configuration management tools.
• Improve system reliability, scalability, troubleshooting efficiency, and operational excellence.

Required Qualifications

• 7+ years of experience in Site Reliability Engineering, Platform Engineering, DevOps, or a related field.
• Hands-on administration experience with Splunk Enterprise or Splunk Cloud.
• Strong proficiency in Splunk SPL, including search development, dashboards, and alerting.
• Practical experience with Elasticsearch/ELK, Prometheus, and Grafana.
• Experience with Grafana Tempo, OpenTelemetry, and distributed tracing.
• Strong understanding of observability across metrics, logs, and traces.
• Experience with Kafka and scalable monitoring or telemetry pipelines.
• Hands-on experience with Terraform and Infrastructure as Code (IaC).
• Programming or scripting experience in Python, Go, Ruby, or Bash.
• Ability to troubleshoot complex distributed systems and improve production reliability.

Preferred Qualifications

• Splunk certification.
• Experience with Kubernetes and cloud platforms such as AWS, Azure, or Google Cloud Platform.
• Experience with Ansible, Consul, CI/CD pipelines, and service mesh technologies.
• Experience supporting FedRAMP-compliant or other regulated environments.

Ideal Candidate: A senior SRE or Observability Engineer who has built and administered enterprise-scale Splunk infrastructure and also has practical experience with modern observability technologies, particularly OpenTelemetry, Grafana Tempo, Prometheus, Elasticsearch, and Terraform.