Apply Now

Senior Splunk Engineer - Infrastructure Operations

• Posted 30+ days ago • Updated 5 hours ago

Full Time

USD 60.00 per day

GovCIO

Fitment

Dice Job Match Score™

📋 Comparing job requirements...

Job Details

Skills

Recruiting
Data Centers
Concurrent Computing
CPU
Extraction
Onboarding
Data Flow
API
RBAC
SPL
Dashboard
Query Optimization
Knowledge Base
IDS
Management
KPI
Capacity Management
Data Retention
Regulatory Compliance
Data Governance
Testing
Data Recovery
High Availability
Root Cause Analysis
Collaboration
ROOT
Orchestration
Cloud Computing
Splunk
Clustering
Security Clearance

Summary

GovCIO is currently hiring for Systems Architect (Senior) /Senior Splunk Engineer - Infrastructure Operations of Infrastructure Operations to support our Administrative Office of the US Courts NLS project. The NLS currently ingest an average of 18-20TB of logging data daily across 60 indexers distributed in 2 data centers. This position is located within the United States and is fully remote.

Responsibilities

Design, implement, and operate the Splunk Core, Enterprise Security, IT Service Intelligence (i.e., ITSI), Phantom (Security Orchestration, Automation, and Response (SOAR)), Splunk Cloud, Splunk On-Call, and Multi-Site Index Clustering environment.
Monitor overall Splunk health through the Monitoring Console (DMC) including indexer, search head, and cluster master status.
Track indexing rates, license usage, queue health, and search concurrency to identify performance or ingestion issues early.
Monitor CPU, memory, and disk utilization across all Splunk components to ensure optimal resource usage.
Respond promptly to health alerts, DMC warnings, or anomalies observed on monitoring dashboards.
Investigate and resolve common user-reported issues such as access problems, failed searches, or non-triggering alerts.
Troubleshoot data ingestion, parsing, and indexing issues across Universal Forwarders, Heavy Forwarders, and HEC endpoints.
Investigate missing or duplicate logs, timestamp errors, or sourcetype misassignments and escalate complex parsing issues to Engineering.
Validate new data source onboardings by confirming sourcetype assignment, timestamp accuracy, and field extraction integrity.
Support data source owners with forwarder deployment, syslog setup, and connectivity troubleshooting during initial onboarding.
Maintain data flow visibility from source forwarder indexer to confirm data completeness and performance.
Rotate and update credentials, API keys, or tokens used in data inputs, integrations, alerts, and scheduled searches.
Manage RBAC user and role mappings, handling access requests, entitlement reviews, and permission troubleshooting.
Provide end-user assistance with SPL searches, reports, alerts, and dashboards, including query optimization tips.
Maintain and update knowledge base articles, SOPs, and FAQs for repeatable issues and troubleshooting steps.
Log and escalate platform or parsing issues to the Engineering team with evidence such as logs, screenshots, and correlation IDs.
Open and manage Splunk Support cases for platform-level bugs, license problems, or critical system faults.
Monitor and manage ITSI service health, including KPIs, correlation searches, NEAP policies, and summary index latency.
Troubleshoot ITSI-related issues such as broken KPIs, delayed episodes, or missing notable events.
Perform capacity management by monitoring index growth, bucket rotation, and frozen data retention policies.
Conduct periodic system maintenance tasks, including orphaned object cleanup and knowledge object review.
Verify and maintain compliance with data governance and retention policies, ensuring secure and auditable configurations.
Participate in DR testing and validation to ensure Splunk data recovery and HA configurations are functioning as expected.
Document incidents, RCA findings, and preventive actions for future reference.
Collaborate closely with the Engineering team for escalations, root-cause investigations, and deployment verifications.

Qualifications

Bachelor's with 10 years (or commensurate experience) OR Masters Degree or higher (in a related discipline) with 7 years experience

Required Skills and Experience

Expert skills in Enterprise Security, ITSI, SOAR, and the Slunk product line.
Able to design, implement, and operate the Splunk Core, Enterprise Security, IT Service Intelligence (i.e., ITSI), Phantom (Security Orchestration, Automation, and Response (SOAR)), Splunk Cloud, Splunk On-Call, and Multi-Site Index Clustering environment.

Clearance Required: Must be able to obtain and maintain AOPublic Trust

Posted Salary Range

USD $105,000.00 - USD $145,000.00 /Yr.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 10443217
Position Id: 7505
Posted 30+ days ago

Company Info

About GovCIO

GovCIO is a rapidly growing provider of advanced technology solutions and digital services for the federal government. Combining our extensive federal experience with the latest innovations in IT and disruptive approaches, our experts develop comprehensive solutions to meet the most pressing demands of today’s government agencies. From the U.S. military to Health and Human Services, we have an impressive track record of helping our customers optimize how they operate.

We’re transforming government IT, empowering our federal customers to meet the challenges of today while building the government of tomorrow.

Go to company profile

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Cribl Engineer

No location provided

•

Today

GovCIO is currently hiring for Data Warehousing Specialist (Network Engineer) of Infrastructure Operations This position will be located within the United States and will be fully remote. Responsibilities Develop apply best practices and tools for data ingestion, indexing, and management to optimize data sources and refine data collection processes to capture only pertinent data. Plan and perform Cribl platform upgrades (Leader, Worker, and Edge nodes) following defined change control procedure

Full-time

USD 125,000.00 - 140,000.00 per year

Incident Handler/Security Analyst - 3rd shift

No location provided

•

Today

GovCIO is currently hiring for a 3rd shift (11pm - 7:30am) Incident Handler/Security Analyst to support the US Courts in Washington, DC (4 days onsite, Fridays remote). Responsibilities Correlates threat data from various sources to establish the identity and modus operandi of hackers active in client's networks and posing a potential threat. Provides the customer with assessments and reports facilitating situational awareness and understanding of current cyber threats and adversaries. Develops

Full-time

USD 81,850.00 - 110,000.00 per year

Senior ServiceNow Enterprise Tools Support Specialist

No location provided

•

Today

GovCIO is seeking a skilled and motivated Senior ServiceNow and Enterprise Tools Support Specialist to support our U Network Operations and Security Center (NOSC) proposal. This role supports mission-critical systems that enhance national security and operational efficiency. The ideal candidate will have extensive experience in managing ServiceNow and enterprise IT tools, offering Tier II/ support, system integration, workflow automation, and service optimization. This position will be in Washin

Full-time

USD 130,000.00 - 140,000.00 per year

Network Senior Engineer / Team Lead

No location provided

•

Today

GovCIO is currently hiring for Systems Senior Engineer(s)/Team Lead. This position will be located in Glynco, GA and will be an onsite position with potential for flexible telework. This role is part of the Federal Law Enforcement Training Centers' (FLETC) Information Technology (IT) Delivery and Operations Support Services (DOSS) program providing the full range of IT services that support FLETC's mission. This work is contingent upon award. #fletc#itdoss Responsibilities Design and implement

Full-time

USD 104,000.00 - 130,000.00 per year

Search all similar jobs