Splunk Engineer

Hi,

Job Title: Splunk Engineer/ Admin

Location: San Jose, CA HYBRID

Duration:-12 months+

Job Description:

• Keeping a multi-site Splunk Enterprise (indexer clustering + SHC) healthy: upgrades/patching, daily/weekly health checks, capacity & license management, DR tests.
• Onboarding data cleanly and securely: forwarders/syslog/HEC; sourcetypes, props/transforms, timestamping/line-breaking, field extractions, retention.
• Improving performance and reliability: monitor ingestion/search performance, queues, storage/bucket health; remove bottlenecks; tune searches and data models.
• Enabling users: create/optimize SPL searches, dashboards, alerts; advise engineers, SREs, and SecOps on best practices and troubleshooting.

The most important duties are

• Operate and harden a multi-site Splunk Enterprise environment (indexer clustering, SHC, deployer/deployment server, RBAC, app lifecycle).
• Monitor and tune ingestion, search, and storage (RF/SF validation; bucket health; NFS tuning; queue depths).
• Lead data onboarding projects across on-prem, SaaS, cloud (Azure/AWS), K8s; ensure auditability and data-handling policy compliance.
• Build/optimize SPL, dashboards, alerts; coach consumers on SPL and performance patterns (tstats, accelerations, base/inline searches).
• Maintain DR posture and execute/verify failovers.

What this job needs to be successful is (traits and characteristics)

• 3 5+ years administering Splunk Enterprise at multi-TB/day scale, including indexer clustering and SHC in multi-site deployments.
• Expert SPL and performance tuning (tstats, data models/accelerations, search optimization).
• Deep data-onboarding skills (forwarders/syslog/HEC) and props.conf/transforms.conf mastery (timestamps, line-breaking, field extraction, value normalization).
• Strong Linux admin + scripting (bash, Python); networking/TLS fundamentals.
• Experience with NFS-backed indexers (operational tuning/gotchas).
• Clear communicator with a customer-enablement mindset; documents well; bias for automation.
• Nice-to-have: Splunk Architect cert; experience with ES, ITSI, MLTK, and SOAR; familiarity with data-science/ML concepts (to partner with teams, not to lead research).

The simplest and easiest way to see that this job is done well is...

• Cluster health green: RF/SF consistently met; successful failover tests.
• Low ingest error rate and low data latency to index; stable license utilization.
• Search KPIs: median and P95 search times within agreed SLOs; reduced scheduler/skipped search rates.
• Clean data: correct timestamps, low unknown sourcetypes, stable field extraction accuracy.
• User outcomes: growing self-service usage, actionable dashboards/alerts, and satisfied internal customers (shorter MTTR for incidents).
• No audit/compliance exceptions related to Splunk data handling or access controls.

Basic qualifications

• 3 5+ years hands-on Splunk Enterprise administration at scale (multi-TB/day), including indexer clustering, SHC, deployer/DS, license mgmt.
• Strong SPL and performance tuning (tstats, DMs, accelerations, base/inline searches).
• Data onboarding expertise: forwarders/syslog/HEC; props/transforms; timestamping/line-breaking; field extractions; retention planning.
• Linux + scripting (bash/Python); networking/TLS fundamentals.
• Experience operating with NFS-backed indexers.
• Nice-to-have: Splunk Architect cert; ES/ITSI/MLTK/SOAR; familiarity with data-science/ML concepts.

Zafar Khan

Technical Recruiter