Senior Splunk Infrastructure Engineer ((Hybrid – Local Candidates Only)

Job Title: Senior Splunk Infrastructure Engineer
Location: Charlotte, NC (Hybrid – Local Candidates Only)
Employment Type: Contract

About the Role

Our client’s Observability team within a leading financial services organization is seeking a Senior Splunk Infrastructure Engineer to support and evolve large-scale logging and monitoring platforms.

This role focuses on maintaining, upgrading, and optimizing enterprise Splunk infrastructure running on Google Cloud Platform (Google Cloud Platform). You’ll collaborate closely with Observability and Security teams to ensure platform reliability, performance, and compliance in a highly regulated fintech environment.

Note: Core Splunk components (Indexers & Search Heads) run on VMs, while Forwarders are deployed in Kubernetes.

Key Responsibilities

• Administer and support multiple Splunk clusters (Indexer Clusters, Search Head Clusters, Heavy Forwarders)

• Maintain and optimizeison Splunk Enterprise Security (ES) infrastructure for security operations

• Manage Heavy Forwarders, data pipelines, parsing rules, and routing logic

• Design and maintain High Availability (HA) and Disaster Recovery (DR) strategies across regions

• Implement safe SaltStack deployment practices (canary testing, validation, staged rollouts)

• Develop and maintain Infrastructure as Code using SaltStack states and formulas

• Provision, monitor, and scale infrastructure on Google Cloud Platform (Google Cloud Platform)

• Perform deep Linux troubleshooting (kernel tuning, disk I/O, memory, networking)

• Participate in on-call rotation to ensure 24/7 platform reliability

• Execute maintenance windows, upgrades, and patching cycles

• Maintain and update runbooks and technical documentation

Required Qualifications

• 5+ years administering large-scale Splunk Enterprise environments

• Strong experience with Indexer Clustering and Search Head Clustering

• Advanced expertise in SPL (Splunk Processing Language)

• Hands-on experience with Splunk Enterprise Security (ES)

• Strong SaltStack experience (custom states, large-scale node management)

• Deep Linux/Unix expertise (RHEL, CentOS, Ubuntu)

• Hands-on experience with Google Cloud Platform (Google Cloud Platform) – GCE & networking

• Proficiency in Python and/or Bash scripting

• Experience supporting production-grade, highly available systems

Preferred Qualifications

• Terraform for infrastructure provisioning alongside SaltStack

• Kubernetes, Helm, and Flux with GitOps workflows

• Platform Engineering / “Done-for-you” infrastructure solutions

• Multi-cloud experience (Google Cloud Platform, AWS, Azure)

• Knowledge of OpenTelemetry (OTEL) and migration from Splunk Forwarders

• Experience working in a regulated fintech environment

Work Authorization & Location Requirements

• Local candidates only – must be able to work hybrid in Charlotte, NC

• Must be authorized to work in the U.S. without sponsorship