Overview
Skills
Job Details
Job Title: Splunk Engineer/ Admin
Location: 3 days a week onsite in San Jose, CA
local to Bay Area candidates only
Job Description:
Keeping a multi-site Splunk Enterprise (indexer clustering + SHC) healthy: upgrades/patching, daily/weekly health checks, capacity & license management, DR tests.
Onboarding data cleanly and securely: forwarders/syslog/HEC; sourcetypes, props/transforms, timestamping/line-breaking, field extractions, retention.
Improving performance and reliability: monitor ingestion/search performance, queues, storage/bucket health; remove bottlenecks; tune searches and data models.
Enabling users: create/optimize SPL searches, dashboards, alerts; advise engineers, SREs, and SecOps on best practices and troubleshooting.
The most important duties are
Operate and harden a multi-site Splunk Enterprise environment (indexer clustering, SHC, deployer/deployment server, RBAC, app lifecycle).
Monitor and tune ingestion, search, and storage (RF/SF validation; bucket health; NFS tuning; queue depths).
Lead data onboarding projects across on-prem, SaaS, cloud (Azure/AWS), K8s; ensure auditability and data-handling policy compliance.
Build/optimize SPL, dashboards, alerts; coach consumers on SPL and performance patterns (tstats, accelerations, base/inline searches).
Maintain DR posture and execute/verify failovers.
What this job needs to be successful is (traits and characteristics)
3 5+ years administering Splunk Enterprise at multi-TB/day scale, including indexer clustering and SHC in multi-site deployments.
Expert SPL and performance tuning (tstats, data models/accelerations, search optimization).
Deep data-onboarding skills (forwarders/syslog/HEC) and props.conf/transforms.conf mastery (timestamps, line-breaking, field extraction, value normalization).
Strong Linux admin + scripting (bash, Python); networking/TLS fundamentals.
Experience with NFS-backed indexers (operational tuning/gotchas).
Clear communicator with a customer-enablement mindset; documents well; bias for automation.
Nice-to-have: Splunk Architect cert; experience with ES, ITSI, MLTK, and SOAR; familiarity with data-science/ML concepts (to partner with teams, not to lead research).
The simplest and easiest way to see that this job is done well is
Cluster health green: RF/SF consistently met; successful failover tests.
Low ingest error rate and low data latency to index; stable license utilization.
Search KPIs: median and P95 search times within agreed SLOs; reduced scheduler/skipped search rates.
Clean data: correct timestamps, low unknown sourcetypes, stable field extraction accuracy.
User outcomes: growing self-service usage, actionable dashboards/alerts, and satisfied internal customers (shorter MTTR for incidents).
No audit/compliance exceptions related to Splunk data handling or access controls.
Basic qualifications
3 5+ years hands-on Splunk Enterprise administration at scale (multi-TB/day), including indexer clustering, SHC, deployer/DS, license mgmt.
Strong SPL and performance tuning (tstats, DMs, accelerations, base/inline searches).
Data onboarding expertise: forwarders/syslog/HEC; props/transforms; timestamping/line-breaking; field extractions; retention planning.
Linux + scripting (bash/Python); networking/TLS fundamentals.
Experience operating with NFS-backed indexers.
Nice-to-have: Splunk Architect cert; ES/ITSI/MLTK/SOAR; familiarity with data-science/ML concepts.