Apple's Services Engineering organization (ASE) is seeking experienced site reliability engineers to build and maintain the workflow orchestration and automation platform for our Data Services fleet. Data Services SRE operates Cassandra, Redis/Valkey, Kafka, Solr, and our Coordination systems (ZooKeeper, etcd, Parallax) across Apple's data centers worldwide. These systems form the platform upon which iCloud and many other internet services at Apple are built. The operations that keep this fleet healthy (rolling upgrades, host replacements, cluster expansions, backup and restore, release qualification) are long-running, multi-step, failure-prone procedures that today rely on internal workflow systems and manual runbooks. You will work to replace these legacy systems with durable, observable, resumable workflows. Your work will reduce on-call toil, cut incident recovery time, and turn tribal operational knowledge into self-documenting code. In ASE, your work benefits hundreds of millions of users and is critical to the reliability of some of the most visible current and future Apple features.
Description
The ASE Data Services workflow effort builds automation that is safe, reliable, observable, and resumable. This work requires an innovative spirit and an extraordinary degree of care and rigor in engineering. These workflows operate on production databases at massive scale, where a mishandled rolling upgrade or host replacement has real customer impact. You will support the workflow engine layer underlying Apples most critical database systems which power all of Apples internet services. You will design and build workers and workflow definitions, composing operations across existing execution layers: host and pod provisioning, cluster topology discovery, and our monitoring and alerting systems. You will migrate operational logic off home-grown workflow tooling onto a durable execution model with retry, signal, query, and full execution history. This role requires excellent communication, the ability to partner closely with developers, SRE, and platform teams, and a high degree of customer focus when engaging with the internal stakeholders who will run these workflows every day. As a distributed team, the ability to work effectively with colleagues based in other locations is essential; experience in this area is a plus. Prior experience building or operating workflow/orchestration systems, or operating distributed databases and storage systems at scale, is recommended.
Minimum Qualifications
BS or MS in Computer Science / related fields or equivalent work experience
5+ years in a Site Reliability Engineering or infrastructure-focused engineering role
Proficient in Java; proficiency in Go (golang) or Python is also valuable
Hands-on experience designing or operating workflow / orchestration / job-scheduling systems, or building substantial automation for stateful, long-running, failure-prone operations
Support of internet-facing production services and distributed systems via deployments, on-call, and incident management
Experience running large-scale infrastructure with a heavy reliance on automation tooling
Real operational experience managing services at scale on Kubernetes
Excellent troubleshooting and performance deep-dive analysis
Self-motivated and inquisitive, with an aptitude to learn new technologies quickly and effectively
Preferred Qualifications
Direct experience with workflow systems (or comparable durable-execution engines such as Temporal or Cadence): writing workflows and activities, reasoning about retries, signals, queries, heartbeating, and execution history
Experience operating or building tooling for distributed databases / storage systems (Cassandra, Redis/Valkey, Kafka, Solr, ZooKeeper, etcd)
Understanding of database concepts: consistency models, isolation levels, crash and recovery semantics
Operational experience deploying in and running on datacenter and cloud architectures: networking topologies, host placement strategies, and failure modes; design of multi-datacenter systems; failure domains; and wide-area networking
Performance engineering (design concepts, profile-guided optimization)
Familiarity with mTLS / certificate-based service auth and Kubernetes-native worker deployment
Experience replacing legacy or home-grown orchestration with modern, observable platforms
Comfort working in a distributed team across multiple sites and time zones
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: 90733111
- Position Id: 8b8c78db7281423b76262fc126cf1644
- Posted 30+ days ago
