Systems Engineer IV - Splunk Enterprise

AVA Consulting is seeking a Systems Engineer IV - Splunk Enterprise

Location: San Jose, CA (Hybrid)

U.S. Citizens and those authorized to work in the U.S. are encouraged to apply. We are unable to sponsor at this time.

Job Description:

• The most important ways the person doing the job should spend their time are:
• Keeping a multi-site Splunk Enterprise (indexer clustering + SHC) healthy: upgrades/patching, daily/weekly health checks, capacity & license management, DR tests.
• Onboarding data cleanly and securely: forwarders/syslog/HEC; sourcetypes, props/transforms, time stamping/line-breaking, field extractions, retention.
• Improving performance and reliability: monitor ingestion/search performance, queues, storage/bucket health; remove bottlenecks; tune searches and data models.
• Enabling users: create/optimize SPL searches, dashboards, alerts; advise engineers, SREs, and SecOps on best practices and troubleshooting.
• Environment (context)
• ~14,000 employees; ~500 active Splunk users
• ~3 TB/day ingest from ~100 sources; NFS-backed storage
• Sources span on-prem apps/appliances/network devices, SaaS, private cloud/K8s, Azure & AWS

Responsibilities:

• Operate and harden a multi-site Splunk Enterprise environment (indexer clustering, SHC, deployer/deployment server, RBAC, app lifecycle).
• Monitor and tune ingestion, search, and storage (RF/SF validation; bucket health; NFS tuning; queue depths).
• Lead data onboarding projects across on-prem, SaaS, cloud (Azure/AWS), K8s; ensure auditability and data-handling policy compliance.
• Build/optimize SPL, dashboards, alerts; coach consumers on SPL and performance patterns (tstats, accelerations, base/inline searches).
• Maintain DR posture and execute/verify failovers.

Requirement:

Required Skills:

• 3 5+ years administering Splunk Enterprise at multi-TB/day scale, including indexer clustering and SHC in multi-site deployments.
• Expert SPL and performance tuning (tstats, data models/accelerations, search optimization).
• Deep data-onboarding skills (forwarders/syslog/HEC) and props.conf/transforms.conf mastery (timestamps, line-breaking, field extraction, value normalization).
• Strong Linux admin + scripting (bash, Python); networking/TLS fundamentals.
• Experience with NFS-backed indexers (operational tuning/gotchas).
• Clear communicator with a customer-enablement mindset; documents well; bias for automation.

Nice-to-have:

• Splunk Architect certification
• Experience with ES, ITSI, MLTK, and SOAR; familiarity with data-science/ML concepts (to partner with teams, not to lead research).

NOTE: Interested Candidates can apply by sending their Updated Resume and Contact Details.

Ron Tolson

AVA Consulting

Fax:

Web: