Systems Engineer II Enterprise Instrumentation Engineer

Systems Engineer II Enterprise Instrumentation Engineer

Join our Enterprise Instrumentation team and play a crucial role in enhancing IT Operations through the implementation of robust monitoring practices for applications across a diverse technology stack. You will leverage Fiserv's approved standard set of monitoring tools, such as Splunk, Dynatrace, CA ASM, Sitescope, Extrahop, Spectrum, CA PM, Moogsoft, and more, to empower operations teams in proactively identifying and remediating issues.

As a senior engineer within the instrumentation team, you will be responsible for the design, development, and implementation of monitoring solutions within complex environments encompassing various technologies. Your expertise will span a wide range of monitoring techniques, including dashboarding, alerting mechanisms, and the creation of custom solutions. By leveraging these techniques, you will ensure that operations teams can effectively monitor the health of applications throughout the organization.

Furthermore, you will have the responsibility of creating insightful dashboards and reports to help technology and business leadership understand the impact of outages on customers. This vital information will aid strategic decision-making processes and provide valuable insights into customer experience.

In addition to implementing monitoring solutions, you will employ machine learning technologies to improve customer experience by reducing the time required to detect, investigate, and resolve issues. Collaborating closely with the leadership team, enterprise architects, technical leads, developers, product owners, account executives, and operations teams, you will contribute to the development and implementation of end-to-end monitoring solutions.

As a mentor, you will guide and support your team members as well as external teams in effectively utilizing and interpreting data from monitoring tools. This knowledge transfer will contribute to building a strong monitoring capability across the organization.

In summary, by joining our Enterprise Instrumentation team, you will have the opportunity to make a significant impact on IT Operations and customer experience. Through the implementation of advanced monitoring solutions, leveraging machine learning technologies, and providing mentorship, you will contribute to the continuous improvement of our monitoring practices and help drive our organization's success.

Responsibilities

As a key member of the Enterprise Instrumentation team, you will have a range of responsibilities aimed at improving system health, efficiency, and accuracy. These responsibilities include:

• Developing end-to-end monitoring dashboards, alerts, and reports using tools such as Splunk, Dynatrace, Extrahop, CA ASM, and Sitescope for both on-premise and cloud applications.
• Exhibiting strong knowledge of Splunk architecture and its various components (indexer, forwarder, search heads, deployment server), heavy and universal forwarder, and the license model.
• Possessing expert-level knowledge and understanding of the SPL language in Splunk and building complex queries to extract insights.
• Applying expertise in implementing summary indexes, KV stores, lookups, and data model acceleration to optimize search performance and reporting within Splunk.
• Managing Splunk data, including onboarding data from various sources such as logs, DB connect, HTTP Event Collector (HEC), and more.
• Creating and deploying Splunk configuration files (inputs.conf, props.conf, transforms.conf) to ensure proper data ingestion, parsing, and transformation according to specific requirements.
• Implementing configurations for line breaking, timestamp extraction and time zone setting, data masking, and data filtering.
• Utilizing Splunk REST APIs for querying and updating data, enabling seamless integration and automation.
• Demonstrating working knowledge of machine learning techniques and applying anomaly detection in real-time for proactive issue identification.
• Onboarding applications onto Dynatrace and creating visibility for processes and services utilizing advanced techniques in Dynatrace.
• Enabling monitoring for cloud container platforms such as Kubernetes, OpenShift, PCF, Azure Kubernetes Services, and Elastic Kubernetes Services.
• Assisting application and support teams in identifying tuning opportunities using Dynatrace.
• Creating advanced scripts (Selenium based) to emulate user behavior in monitoring tools such as CA ASM, Sitescope, and Dynatrace for synthetic monitoring efforts.
• Having working knowledge of programming and scripting languages (e.g., C, Python, Selenium, Shell etc) to support automation and customizations.
• Working in an Agile environment with multiple sprints, collaborating effectively with cross-functional teams.
• Being process and detail-oriented, with excellent communication, collaboration, documentation, time management, and task completion skills.
• Adapting to new and evolving corporate IT and security lifecycle guiding principles to ensure compliance and alignment.

Basic Qualifications

• Bachelor's or Master's degree in Computer Science, a related technical or scientific field, or equivalent experience.
• 10+ years of experience in implementing monitoring and performance engineering practices, showcasing your expertise in this domain.
• 10+ years of scripting or software development experience, highlighting your proficiency in developing scripts or software solutions.
• 6+ years of experience working with large-scale financial systems architectures, demonstrating your understanding of complex system environments.
• 7+ years of experience using monitoring tools such as Splunk, Dynatrace, CA ASM, Sitescope, NPM tools (e.g., Extrahop), or equivalent tools.
• Proven ability to thrive in a fast-paced, cross-functional team environment, with at least 10+ years of demonstrated successful collaboration and teamwork.

Certifications

• Splunk 9.x Power User and Splunk 9.x Admin Certifications are required.
• Splunk Architect certification is optional.