When you join VerizonYou want more out of a career. A place to share your ideas freely - even if they're daring or different. Where the true you can learn, grow, and thrive. At Verizon, we power and empower how people live, work and play by connecting them to what brings them joy. We do what we love - driving innovation, creativity, and impact in the world. Our V Team is a community of people who anticipate, lead, and believe that listening is where learning begins. In crisis and in celebration, we come together - lifting our communities and building trust in how we show up, everywhere & always. Want in? Join the #VTeamLife.
What you'll be doing...As a Distinguished Engineer- Cloud, you will independently lead requirements, analysis, design and development effort for the observability components of this 80,000 server platform. The core of this role is owning VCP's metrics and log observability platform end to end - the responsibilities below for hardware monitoring, secrets management, and documentation are real and substantial, but they connect to and support that central platform rather than standing apart from it.
We engineer, build, and operate a global private cloud infrastructure called Verizon Cloud Platform (VCP) powered by OpenStack, Kubernetes, and NetApp that supports applications used for numerous internal and external Verizon networks.
You will be responsible for the following but not limited to:
- Owning the architecture and day-to-day operation of the VCP infrastructure metrics and log observability platform - including a dedicated multi-zone VictoriaMetrics cluster purpose-built for long-term capacity planning across OpenShift and NetApp storage, and an active initiative consolidating multiple independent log clusters into a single, cross-site-replicated logging platform.
- Designing and maintaining the recording-rule pipelines, cross-repo metric filtering, and dashboarding that keep that data accurate and trustworthy at scale - work that requires real quantitative judgment: choosing the right aggregation method for a given signal, reasoning about metric cardinality and its memory and cost impact, and validating capacity assumptions against actual usage data rather than spec-sheet numbers.
- Owning and managing a highly capable set of systems in the lab to model production and to monitor development lab systems.
- Designing and building automation and internal tooling from scratch where off-the-shelf options don't fit - including custom services that extract live infrastructure state (Kubernetes, physical network topology, hardware inventory) and model it for cross-system querying and analysis.
- Finding, comparing and testing solutions to new problems or alternatives to existing solutions, whether they be off-the-shelf free or purchased products or internally developed custom solutions. You should be prepared to collaborate with stakeholders in the decision-making and to advocate for your chosen solution using real data and analysis.
- Providing active and assertive leadership on key projects designed to simplify processes, reduce expenses, free up capital, and drive towards the network architecture of the future.
- Engineering of OpenShift/Kubernetes (architecture, design and administration) - including managing GitOps-based deployment pipelines (ArgoCD) across multiple production OpenShift-base Webscale clusters spanning core, edge, and faredge environments.
- Leading secrets-management architecture and migrations (e.g., moving legacy Vault-integration patterns to modern External Secrets Operator approaches) across a multi-cluster fleet.
- Owning monitoring and automation for bare metal hardware health and lifecycle data (Redfish/BMC), in partnership with the team's hardware leads.
- Developing documentation, automation and test plans to support test, deployment and operations teams.
- Responding to and driving root-cause resolution for production incidents affecting the metrics/logging platform itself.
What we're looking for...
We're looking for someone who investigates and learns new technologies independently, not just someone who executes a known playbook.
You'll need to have: - Bachelor's degree or four or more years of work experience.
- Eight or more years of relevant experience required, demonstrated through work and/or military experience, or specialized training.
- Production experience administering Kubernetes clusters at scale (OpenShift or vanilla Kubernetes).
- Experience with Prometheus-compatible metrics systems such as Prometheus, Thanos, Cortex, Victoriametrics or similar - including recording rules, federation, and multi-cluster/multi-tenant architectures.
- Experience in Log management using tools like ELK, Splunk, Victorialogs or other notification systems like Icinga or Alertmanager.
- Experience in software development experience in Python.
Even better if you have one or more of the following:- Bachelor's Degree or experience in Computer Science, Mathematics, or other degree focused on problem-solving
- Experience in operating network or infrastructure monitoring at true carrier/hyperscale - tens of thousands of nodes or more, not the hundreds-to-low-thousands typical of large enterprise environments.
- Experience in log collection cycle, common in enterprise monitoring.
- Experience in software development in Python (or equivalent), involving building and maintaining standalone services and not limited to writing automation scripts.
- Knowledge of cloud-native application architecture and cloud platforms.
- Knowledge of NFV and SDN products, technologies, architectures, and vendors
- OpenStack and Kubernetes certification-level technical knowledge and expertise.
- Experience in Public, Private, and/or Hybrid Cloud processes.
- Development experience using GoLang
- Experience with deployment or use of configuration management tools such as; Ansible, Helm, or Argocd
- GitOps experience with ArgoCD, managing app-of-apps patterns across multiple production Kubernetes/OpenShift clusters.
- Proficiency with Git-based version control workflows, particularly GitLab, at production scale.
- Direct experience with VictoriaMetrics and/or VictoriaLogs specifically, distributed cluster operation, multi-tenancy, and capacity/retention planning at multi-terabyte-per-day ingest scale.
- Experience in Storage platform - NetApp, MinIO, or other enterprise/object storage systems.
- Grafana dashboard design and broader observability/visualization best practices.
- Experience with Kafka and stream-processing pipelines in an operational monitoring context.
- Experience with graph databases (NebulaGraph, Neo4j, or similar) for modeling infrastructure or network topology.
- Familiarity with Redfish/BMC-based hardware monitoring and baremetal fleet management.
- Experience with External Secrets Operator, HashiCorp Vault, or comparable secrets-management platforms in Kubernetes environments.
- Familiarity with OIDC/identity federation patterns (Dex, Keycloak, or similar) for securing internal tooling access.
- Possess quantitative and statistical reasoning - choosing the right aggregation method for a given signal, reasoning about metric cardinality and its memory/cost impact, and validating capacity assumptions against real usage data rather than spec-sheet numbers.
If Verizon and this role sound like a fit for you, we encourage you to apply even if you don't meet every "even better" qualification listed above.
Where you'll be workingIn this hybrid role, you'll have a defined work location that includes working from home and a minimum of three days per week in the office, which will be set by your manager. Employees are responsible for maintaining compliance with hybrid work policies.
Scheduled Weekly Hours40
Equal Employment OpportunityVerizon is an equal opportunity employer. We evaluate qualified applicants without regard to veteran status, disability or other legally protected characteristics.
Benefits and CompensationOur benefits are designed to help you move forward in your career, and in areas of your life outside of Verizon. From health and wellness benefit options including: medical, dental, vision, short and long term disability, basic life insurance, supplemental life insurance, AD&D insurance, identity theft protection, pet insurance and group home & auto insurance. We also offer a matched 401(k) savings plan, up to 8 company paid holidays per year and up to 6 personal days per year, paid parental leave, adoption assistance and tuition assistance, plus other incentives, we've got you covered with our award-winning total rewards package. Depending on the role, employees have the opportunity to receive compensation in the form of premium pay such as overtime, shift differential, holiday pay, allowances, etc. Newly hired employees receive up to 15 days of vacation per year, which grows with additional service. For part-timers, your coverage will vary as you may be eligible for some of these benefits depending on your individual circumstances.
The salary will vary depending on your location and confirmed job-related skills and experience. This is an incentive based position with the potential to earn more. For part-time roles, your compensation will be adjusted to reflect your hours.
The salary range for the location(s) listed on this job requisition based on a full-time schedule is: $137,000.00 - $263,000.00.
.png?format=webp)
