Overview
Skills
Job Details
Job Title: Splunk Engineer/ Admin
Duration: 3 months
Location: Onsite -- San Jose, CA 95125
Environment (context)
~14,000 employees; ~500 active Splunk users
~3 TB/day ingest from ~100 sources; NFS-backed storage
Sources span on-prem apps/appliances/network devices, SaaS, private cloud/K8s, Azure & AWS
Job Description:
Keeping a multi-site Splunk Enterprise (indexer clustering + SHC) healthy: upgrades/patching, daily/weekly health checks, capacity & license management, DR tests.
Onboarding data cleanly and securely: forwarders/syslog/HEC; sourcetypes, props/transforms, timestamping/line-breaking, field extractions, retention.
Improving performance and reliability: monitor ingestion/search performance, queues, storage/bucket health; remove bottlenecks; tune searches and data models.
Enabling users: create/optimize SPL searches, dashboards, alerts; advise engineers, SREs, and SecOps on best practices and troubleshooting.
The most important duties are
Operate and harden a multi-site Splunk Enterprise environment (indexer clustering, SHC, deployer/deployment server, RBAC, app lifecycle).
Monitor and tune ingestion, search, and storage (RF/SF validation; bucket health; NFS tuning; queue depths).
Lead data onboarding projects across on-prem, SaaS, cloud (Azure/AWS), K8s; ensure auditability and data-handling policy compliance.
Build/optimize SPL, dashboards, alerts; coach consumers on SPL and performance patterns (tstats, accelerations, base/inline searches).
Maintain DR posture and execute/verify failovers.
What this job needs to be successful is (traits and characteristics)
3 5+ years administering Splunk Enterprise at multi-TB/day scale, including indexer clustering and SHC in multi-site deployments.
Expert SPL and performance tuning (tstats, data models/accelerations, search optimization).
Deep data-onboarding skills (forwarders/syslog/HEC) and props.conf/transforms.conf mastery (timestamps, line-breaking, field extraction, value normalization).
Strong Linux admin + scripting (bash, Python); networking/TLS fundamentals.
Experience with NFS-backed indexers (operational tuning/gotchas).
Clear communicator with a customer-enablement mindset; documents well; bias for automation.
Nice-to-have: Splunk Architect cert; experience with ES, ITSI, MLTK, and SOAR; familiarity with data-science/ML concepts (to partner with teams, not to lead research).
The simplest and easiest way to see that this job is done well is
Cluster health green: RF/SF consistently met; successful failover tests.
Low ingest error rate and low data latency to index; stable license utilization.
Search KPIs: median and P95 search times within agreed SLOs; reduced scheduler/skipped search rates.
Clean data: correct timestamps, low unknown sourcetypes, stable field extraction accuracy.
User outcomes: growing self-service usage, actionable dashboards/alerts, and satisfied internal customers (shorter MTTR for incidents).
No audit/compliance exceptions related to Splunk data handling or access controls.
Basic qualifications
3 5+ years hands-on Splunk Enterprise administration at scale (multi-TB/day), including indexer clustering, SHC, deployer/DS, license mgmt.
Strong SPL and performance tuning (tstats, DMs, accelerations, base/inline searches).
Data onboarding expertise: forwarders/syslog/HEC; props/transforms; timestamping/line-breaking; field extractions; retention planning.
Linux + scripting (bash/Python); networking/TLS fundamentals.
Experience operating with NFS-backed indexers.
Nice-to-have: Splunk Architect cert; ES/ITSI/MLTK/SOAR; familiarity with data-science/ML concepts.
Rate/Salary: A reasonable estimate of the current pay range for this position is $55.00 to $60.00 hourly. Actual pay will be based on a variety of factors, including shift, location, experience, skill set, performance, licensure and certification, and business needs, and will be set by your employer. The range for this position in other geographic locations may differ. Certain positions may also be eligible for variable incentive compensation, such as bonuses or commissions, that is not included in base pay.
If you have any questions or concerns about this posting, please email SRpostings at wwt.com