Position Summary The
Principal Engineer is the lead architect and hands-on builder of our unified AI Platform as a Service. This role is responsible for transforming raw foundation model capabilities into a scalable, multi-tenant reasoning stack that empowers the entire enterprise to build, deploy, and manage semantic discovery, conversational intelligence, and autonomous agents. This role will balance 40% hands-on systems development with 60% platform strategy, personally coding the core orchestration engines, standardized capability servers, and universal trust guardrails. The mission is to provide a central 'AI Operating System' for the company, ensuring that specialized agents across different business units can communicate via inter-agent protocols, access grounded knowledge layers securely, and execute autonomous tasks within a governed, high-performance agent runtime environment.
The Principal Engineer in Digital Ecommerce at Costco holds a pivotal leadership role, steering the strategic vision, design, development and architecture of high performance native mobile applications for both iOS and Android platforms. In this capacity, this role will be instrumental in shaping the long-term technical strategy, embracing innovation, and providing hands-on experience as a technical leader for the teams supporting these platforms; conducting research for emerging industry trends and providing ongoing guidance to bring that to fruition. This role demands a dynamic approach, blending direct technical guidance, mentorship, and active participation in risk and growth management. Collaboration with the Central Engineering leadership team is integral, fostering a hands-on and collective effort towards achieving organizational goals.
As a Principal Engineer, you will be instrumental in ensuring that our mobile solutions seamlessly integrate with microservices architectures and ContentStack for dynamic content management.
In this role, you'll work with cross-functional teams to build scalable, secure, and innovative mobile solutions while driving technical excellence within the team. You'll be responsible for optimizing mobile app performance, implementing CI/CD processes, and ensuring the scalability of mobile systems.
Job Duties/Essential Functions- Leads the architecture and development of native mobile applications for iOS (Swift/Objective-C) and Android (Kotlin/Java).
- Designs efficient and scalable mobile app architectures that integrate seamlessly with microservices and backend systems (RESTful APIs, GraphQL, etc.).
- Focuses on optimizing networking, memory management, and UI responsiveness for both platforms to ensure high performance and low latency.
- Works with microservices architecture to build mobile solutions that communicate with backend services through REST, gRPC, or GraphQL APIs.
- Collaborates with backend teams to design API contracts and ensure mobile apps are optimized for asynchronous communication and real-time data sync.
- Architects and implements mobile applications that integrate with ContentStack (or similar headless CMS platforms) to manage content dynamically.
- Ensures that content delivery from ContentStack is seamless, responsive, and optimally synchronized with mobile devices.
- Establishes and enforces high standards for clean code, modular architecture, and best practices in mobile development.
- Leads code reviews, provides feedback, and mentors other engineers to improve technical skills and maintain consistent coding standards.
- Oversees the use of automated testing, including unit tests (JUnit/XCTest), integration tests, and UI automation to ensure mobile apps are robust and reliable.
- Profiles and optimizes mobile apps for speed, memory efficiency, and battery consumption.
- Uses mobile-specific profiling tools such as Xcode Instruments, Android Profiler, and network monitoring tools to identify and resolve performance bottlenecks.
- Designs, implements, and optimizes CI/CD pipelines for mobile applications using tools.
- Automates the build, test, and release processes to ensure apps are always production-ready and updates are delivered efficiently.
- Provides leadership to mobile engineers, guiding them in best practices and advanced development techniques.
- Sets the technical direction for the mobile team, ensuring alignment with overall company goals and architectural standards.
- Mentors junior and senior engineers, providing them with opportunities to grow and improve their technical skills.
Ability to operate vehicles, equipment or machinery.- Computer, phone, printer, copier, fax
Non-Essential Functions- Assists in other areas of the department as necessary.
- Assists in other areas of the company as necessary.
Ability to operate vehicles, equipment or machinery.- Same as essential functions
Experience, Skills, Education & Licenses/CertificationsRequired:- 15+ years of experience in native mobile development (iOS: Swift/Objective-C, Android: Kotlin/Java) with a deep understanding of mobile architecture.
- 10+ years' experience in leadership or senior roles in mobile ; demonstrating the ability to guide and lead a team effectively.
- 10+ years in Software Engineering, with at least 4 years in a Principal or Architect-level role.
- 15+ years of experience working with microservices architectures, RESTful APIs, gRPC, and integrating mobile applications with backend services.
- Minimum 15 years of experience as a mobile developer.
- Should have worked in retail or banking in mobile application development projects.
- Strong expertise with headless CMS platforms like ContentStack, including API integrations, content delivery, and synchronization.
- Expertise in mobile app optimization techniques, including memory management, UI responsiveness, and network efficiency.
- Solid experience with unit testing (JUnit/XCTest), integration testing, and UI automation for mobile apps.
- Proven experience leading technical teams, influencing architectural decisions, and mentoring engineers.
- Demonstrated advanced proficiency in API management tools, API development, platform and services security (e.g., Apigee,DataPower, Springboot, SQL, Java, C#)
- Proven expertise in working with cloud platforms, such as AWS, Azure or Google Cloud, to architect and implement scalable and efficient platforms and services.
- Experience dealing with high-volume and high-velocity platforms and services, and designing systems that can handle the demands of real-time or near-real-time operations.
- Demonstrated ability to take ownership and responsibility of critical issues and conclude them to resolution.
- Strong problem-solving and decision-making abilities.
- Certified Google Cloud Professional Cloud Architect
- Experience leading technical workstreams, translating business problems into AI-native architectures.
- 2+ years specifically architecting LLM-based systems, with a proven track record of moving agentic projects into production at scale.
- Expertise in asynchronous orchestration frameworks (e.g., Python) and proficiency in statically typed systems (e.g., Java, Go, or Rust) is required to engineer high-concurrency agentic middleware using stateful graph orchestration (e.g., ADK or LangGraph) to power robust, autonomous reasoning engines.
- Expertise in cloud-native CLI tools and Infrastructure-as-Code frameworks for automating agentic infrastructure deployment. Proven track record of deploying and scaling containerized autonomous workloads using enterprise-grade container orchestration and serverless execution platforms.
- Experience managing high-scale distributed architecture, vector databases, graph databases and structured data pipelines.
- Deep knowledge of stateful orchestration frameworks and multi-agent design patterns, with the architectural expertise to engineer custom reasoning engines and proprietary orchestration logic when off-the-shelf solutions reach their scaling or safety limits.
- Practitioner understanding of Chain-of-Thought, ReAct, Tree-of-Thoughts, and Self-Reflection architectures.
- Experience managing systems with millions of daily requests or handling multi-petabyte datasets.
- Proficiency in architecting semantic retrieval layers, attribute-aware discovery, and stateful persistence systems to provide high-fidelity long-term context for autonomous agents.
- Deep understanding of MCP, A2A, REST/gRPC APIs, Oauth2 security, and function calling mechanics.
- Familiarity with design patterns and microservices-based architecture patterns.
- Mastery of distributed traceability, neural telemetry, and cognitive debugging suites to audit and visualize logic trajectories across complex inter-agent handoffs.
- Understanding of global AI regulations (e.g., EU AI Act) and how to translate them into technical guardrails.
- 5+ years of experience developing within an agile methodology.
- Strong verbal and written communication skills and be able to communicate to both technical and Business audiences.
- Ability to work under pressure in crisis with a strong sense of urgency.
- Responsible, conscientious, organized, self-motivated and able to work with limited supervision.
- Detail-oriented and possess strong problem-solving skills and ability to analyze potential future issues.
- Able to support off-hours work as required, including weekends, holidays, and 24/7 on-call responsibilities on a rotational basis.
Recommended: - Bachelor's or Master's Degree in Computer Science, Informatics, Data science or related technical discipline.
- Cross-Platform Development: Experience with Kotlin Multiplatform Mobile (KMM) or other cross-platform frameworks for mobile app development.
- Cloud & Serverless: Familiarity with cloud-native mobile development, serverless architectures, and using AWS, Google Cloud Platform, or Azure for backend integration.
- Real-Time Communication: Knowledge of WebSockets or MQTT for implementing real-time data synchronization between mobile apps and backend services.
- Security: Familiarity with OAuth 2.0, JWT, encryption, and other mobile security best practices to ensure data protection.
- Open Source Contributions: Contributions to open-source projects related to mobile development or ContentStack integrations.
- Relevant industry certifications are a plus.
- Experience in retail, manufacturing, ecommerce, supply chain.
- Certified Google Cloud Professional Machine Learning Engineer or any agentic AI specialty certification focusing on multi-agent Systems and autonomous reasoning.
- 3+ years distributed cache technologies
- Experience with deploying and configuring Cloud Platform resources.
- Experience working in a retail ecommerce environment.
- Knowledge of: Business ecosystems, SaaS, infrastructure as a service (IaaS), platform as a service (PaaS), SOA, APIs, open data, microservices, event-driven IT, predictive analytics, Cloud Networking, Data Security, Infrastructure Security, Zero Trust.Security Monitoring, Identity and Access Management.
- Familiarity with enterprise architecture tools, related graphical models, and frameworks
- Analytical, planning and organizational skills. Organizationally savvy, with situational and contextual intelligence of the political climate of the enterprise and how to navigate obstacles and politics
- Experience with Public Cloud services (Microsoft Azure, Google Cloud Platform), Cloud Security knowledge.
- Experience with automation and orchestration of infrastructure operational processes (Ansible, BMC, VMware, Red Hat, Chef, Puppet, etc.).
- Experience with Containers (Kubernetes, Docker, etc.).
- Experience working in ERP and Retail systems software (IBM i and/or SAP), including knowledge of Product Lifecycle Management.
- Experience with distributed Enterprise Architecture.
- Architecture experience with COTS package integration experience
- Architecture Professional Certification (e.g., IASA CITA-F, CITA-A or higher).
- Proficient in services integration, messaging, object-oriented programming concepts, multi-tiered distributed component-based technology, legacy infrastructure platforms and technologies, and enterprise service bus.
- Other architecture certifications (e.g. TOGAF 9 certification, SEI).
- Working knowledge of DevOps, agile methodology, and continuous improvement.
- Experience and understanding of Costco's business model and legacy systems.
- Analytical thinker possessing strong problem solving and decision-making skills.
- Proven experience with creating documentation for communication with varied audiences.
- Deep understanding of business requirements elicitation pursuant to specific programs/projects involved.
- Proficient in Google Workspace applications, including Sheets, Docs, Slides, and Gmail.
- Successful internal candidates will have spent one year or more on their current team.
Other Conditions- Management will review the Job Analysis for this position prior to a job offer.
Required Documents Cover Letter
Resume
Last two performance reviews
Attendance records for current year (Do not include absences covered by paid sick/personal time,
FMLA or other protected absences.)
California applicants, please click here to review the Costco Applicant Privacy Notice.
Never miss an opportunity! Create an alert based on the job you applied for.
