Role: Principal Engineer - IoT systems
Location: Remote. 50% travel to HQ, engineering sites, or customer and distribution locations.
ABOUT US
Client is a market leader in developing and delivering innovative healthcare solutions including medication management, mobile computing, and pharmacy automation. We are on a strategic mission to integrate digital capabilities and enhance product performance through robust enterprise architecture and R&D-driven innovation.
_________________________________________________________________________
ABOUT THIS ROLE
We are looking for a Principal Engineer with experience building IoT systems: software that sends commands to physical equipment and handles the responses that come back. At Client, that equipment includes conveyors, diverters, programmable logic controllers (PLCs), vision systems, and label printers in our pharmacy automation and central fill platforms. You will own the .NET services and messaging layers that control this equipment, along with the integrations to the pharmacy systems that send us work. This is a hands-on senior engineering role: you will set technical direction and mentor other engineers, as a Middleware Principal you will own device communication architecture.
The work is real-time and the results are physical. When our software tells a machine to move a container, the message travels over MQTT or gRPC to a controller on the plant floor, the machine acts on it, and a response comes back that our software has to match the original request. This happens in milliseconds, across many machines at many customer sites.
Real environments are messy. Networks drop, controllers reboot, and responses arrive late, out of order, or twice. In a typical web application, repeated messages are a minor problem. Here it can mean a second diverter command or a duplicate label, so the system has to recognize a message it already handled and do nothing. That is why guaranteed delivery and idempotency matter in this role, along with security: the same paths can carry protected health information (PHI). The platform also runs three ways: installed at a customer site, in a customer's cloud, and as a multi-tenant service Client hosts. If you enjoy making distributed systems behave when the network and the hardware work against you, this is a good fit.
Key Responsibilities
Platform Architecture and Distributed Systems
- Own the design and implementation of core platform services in .NET and C#: service boundaries, domain models, transactional consistency, and the failure semantics that decide what happens when a device, a dependency, or a database does not answer.
- Keep the architecture layered so device protocols, business rules, and persistence stay independently testable and deployable, and lead the incremental decomposition of tightly coupled components where the operational payoff is real rather than theoretical.
- Set direction for state management, distributed transactions, outbox patterns, and eventual consistency, making the consistency model explicit for each workflow instead of leaving it to whichever library happened to be used.
- Design for one codebase that runs in cloud, on customer premises, and at the edge, so deployment differences stay in configuration rather than in forked logic.
Messaging, IoT, and Device Communication
- Design and build the messaging backbone: Kafka for durable event streaming, gRPC for low-latency service calls, and MQTT for device and controls communication, owning topic and partition design, consumer semantics, ordering guarantees, quality of service levels, retention, and backpressure.
- Own the IoT communication architecture for a distributed fleet of machines, controllers, and workstations across many customer sites: protocol trade-offs (MQTT, AMQP, gRPC streaming, OPC UA, existing industrial interfaces), device identity, provisioning and lifecycle management, and twin or shadow state with desired and reported reconciliation after reconnection.
- Design for intermittent connectivity and fleet scale: store and forward buffering, local autonomy so a packaging line keeps running when the cloud is unreachable, controlled catch-up on reconnect, keepalive and last will handling, reconnect storms, backoff with jitter, telemetry sampling and cardinality discipline, and correct handling of clock skew and out-of-order arrival.
- Engineer command and control over asynchronous transport (correlation identifiers, acknowledgement semantics, timeouts, cancellation, stale command expiry, compensating action), govern schema and contract versioning across mixed firmware with staged rollout and rollback, and modernize enterprise integrations with pharmacy, order management, ERP, and analytics systems using API-first and event-driven patterns.
Guaranteed Delivery, Idempotency, and State Consistency
- Choose and defend delivery semantics per flow (at-most-once, at-least-once, effectively-once), and implement guaranteed delivery through durable persistence, MQTT quality of service and session decisions, Kafka acknowledgement and replication settings, offset commit discipline, and acknowledgement only after the work is durably recorded.
- Design idempotent consumers and command handlers using idempotency keys, deduplication windows, conditional writes, optimistic concurrency, and natural business keys, so a duplicate device message never causes a duplicate physical outcome such as a second diverter command, a second label print, or a double carrier movement.
- Apply transactional outbox and inbox, saga patterns, deliberate retry with poison message detection and dead letter handling, ordering only where the workflow requires it, and reconciliation and drift detection between platform state and physical device state, proven by duplicate injection, message loss, broker failover, network partition, and device restart tests.
Low-Latency, High-Throughput Concurrent Processing
- Engineer high-throughput concurrent processing in C# using async and await correctly at scale, channels, pipelines, task scheduling, thread pool tuning, and low-contention data structures, and set and defend latency budgets so regressions surface in continuous integration rather than on a customer packaging line.
- Profile and optimize allocation, garbage collection pressure, serialization, and data access using tools such as dotnet-counters, dotnet-trace, PerfView, and BenchmarkDotNet, and build load, soak, and chaos harnesses that reproduce peak throughput and degraded device conditions before release.
Multi-Tenancy, Containerization, and Deployment
- Design tenant isolation across data, messaging, compute, configuration, and secrets, with boundary rules that make cross-tenant access structurally impossible rather than policy-dependent, plus tenant-aware routing, rate limiting, noisy-neighbor protection, and per-tenant capacity and cost visibility.
- Build config-driven tenant and site onboarding, provisioning, and device registration so adding a customer is an operation rather than an engineering project, and design migration paths that move existing single-tenant deployments onto the multi-tenant platform without disrupting live pharmacy operations.
- Own container and Kubernetes strategy for cloud and constrained on-premise or edge environments: hardened images, resource limits, graceful shutdown for in-flight device commands, Helm packaging, GitOps delivery, infrastructure as code, canary deployment and rollback, disaster recovery, and pipelines covering automated testing, security scanning, and software bill of materials generation.
Observability, Monitoring, and Operational Resilience
- Instrument the platform with Open Telemetry so a message can be followed from a device event through every service and back to a device command, with structured logging, metrics, per-device and per-site health scoring, and the dashboards, service level objectives, and alerting that let support teams see a degraded packaging line before the customer calls.
- Design health checks, circuit breakers, retry and dead letter handling, and fault isolation so a single failing device, integration, or tenant does not degrade the platform, and lead incident response and the corrective engineering that prevents recurrence.
Device, API, and Service-to-Service Security
- Own device-to-cloud and device-to-API security: per-device identity, certificate-based authentication with lifecycle and rotation, mutual Transport Layer Security, revocation of compromised devices, and authorization scoped so a controller or line can act only within its own scope.
- Secure the MQTT layer through broker authentication, topic-level access control, tenant and site scoping of topic hierarchies, payload integrity, and protection against replay and command injection on downlink paths.
- Establish service-to-service security across microservices using mutual Transport Layer Security or a service mesh, short-lived workload identity, audience-scoped least-privilege claims, and no shared long-lived credentials; define API security standards for REST and gRPC (OAuth 2.0, OpenID Connect, token and scope enforcement, input validation, rate limiting) with a Zero Trust posture that treats the plant network as untrusted, and own secrets and key management with centralized vaulting and automated rotation.
PHI Protection and Regulatory Compliance
- Design PHI handling to HIPAA Security Rule expectations: encryption in transit and at rest, access controls, unique user and service identification, integrity controls, tamper-evident audit logging of PHI access, and data minimization through de-identification, tokenization, or field-level encryption so PHI reaches only the services that require it.
- Keep PHI out of logs, traces, metrics, error payloads, dumps, and non-production environments through automated checks; define retention, secure deletion, backup encryption, and disaster recovery consistent with business associate agreement obligations; and embed HIPAA, HITRUST, NIST Cybersecurity Framework, and medical device cybersecurity expectations into design from the start with threat modeling, software bill of materials, vulnerability management, and audit-ready traceability.
AI-Assisted Engineering and Developer Productivity
- Use Claude Code as a working part of the daily engineering toolchain for implementation, refactoring, test generation, code review, debugging, and documentation, and demonstrate the practice credibly to other engineers.
- Build the internal scaffolding that makes AI-assisted development effective and safe in a regulated setting (repository and project context files, custom commands, subagents, skills, hooks, and Model Context Protocol integrations), and establish guardrails for AI-generated code covering human accountability, traceability, licensing hygiene, secret handling, and validation expectations for regulated software.
- Measure where AI assistance genuinely improves cycle time, and prepare the platform for AI-enabled operational capabilities by exposing clean APIs, event streams, and telemetry that future monitoring, predictive maintenance, and workflow optimization agents can consume.
Engineering Excellence and Technical Leadership
- Raise the bar through design reviews, threat modeling, code review standards, automated quality gates, and test strategy covering unit, integration, contract, and hardware-in-the-loop testing, with static and dynamic analysis, dependency and container scanning, and secret detection in the pipeline.
- Mentor senior and staff engineers on distributed systems, concurrency, performance engineering, and production ownership; write the architecture decision records and reference implementations that make good patterns the path of least resistance; and make technical risk and operational impact legible to product and executive stakeholders.
To Be Successful in This Role, You Will Need:
- 12 or more years of professional software engineering experience, with 5 or more years building distributed, real-time, or high-throughput production systems.
- Deep, current, hands-on expertise in .NET and C#, including modern async programming, performance tuning of production services, and designing and operating microservices or modular architectures at production scale.
- Production experience with gRPC, and MQTT, with a working understanding of delivery guarantees, ordering, and backpressure in each.
- Substantial IoT or connected device platform experience: device identity and provisioning, twin or shadow state, fleet-scale connection management, offline and intermittent connectivity, telemetry pipelines, and over-the-air configuration or firmware rollout.
- Deep practical knowledge of guaranteed delivery and idempotency: at-least-once and effectively once processing, deduplication, transactional outbox, partitioning and ordering, retry and dead letter handling, and reconciliation of divergent state.
- Demonstrated results in low-latency, high-throughput concurrent processing, with specific examples you can walk through.
- Experience designing multi-tenant SaaS platforms, strong containerization and Kubernetes practice with infrastructure as code and continuous delivery, and practical command of observability tooling including Open Telemetry and service level objectives.
- Demonstrated depth in application and platform security as a practitioner rather than a reviewer: device and machine identity, certificate-based authentication and mutual Transport Layer Security, OAuth 2.0 and OpenID Connect, service-to-service authorization, MQTT broker and topic-level access control, secrets management, and threat modeling.
- Working experience using Claude Code or comparable AI coding tools as part of a professional engineering workflow, with a clear view of where they add value and where they need supervision.
- Ability to move between writing production code and framing an architectural trade-off for executives, with the judgment to choose the simple design when it is sufficient.
Preferred Qualifications
- Experience in healthcare, medical device, pharmacy, or industrial automation, or another regulated domain.
- Direct experience integrating software with PLCs, conveyors, vision systems, or label printers.
- Experience modernizing an installed base of customer-deployed software toward a SaaS and multi-tenant model without disrupting live operations.
- Familiarity with hybrid and edge constraints: limited connectivity and customer-controlled infrastructure.
- Experience with , AWS IoT Core, Azure IoT Hub, or operating an MQTT broker fleet directly.
Location: Remote. 50% travel to HQ, engineering sites, or customer and distribution locations.
Never miss an opportunity! Create an alert based on the job you applied for.