Senior Unix/Linux System Administrator
Contract right now is till the edge if Jan will most likely be extended
100% Remote
Not just any System Admin but specifically within Azure IOT edge
Must haves
- Azure IoT Edge
- Great Scripting
- Observability
- Virtual Machines
- LINUX-RHEL
Job Summary
We are looking for an experienced Unix/Linux System Administrator with strong systems automation experience to administer, operate, and automate our Linux-based server and edge computing environments, including Azure IoT Edge devices, at scale across a fleet of approximately 11,000 virtual machines. This role is responsible for day-to-day system administration, patching, automation, container operations, and production-grade observability (OpenObserve, PrometheGrafana) across on-premises and cloud-based disaster recovery systems. The ideal candidate is a hands-on operator who is comfortable scripting, troubleshooting, and maintaining CI/CD pipelines in a fast-paced, always-on environment.
Experience Required
10+ years of hands-on experience as a Unix/Linux System Administrator, with a strong track record of operating production Linux systems, containers, and CI/CD-driven automation workflows at scale.
Key Responsibilities
- Administer and support Azure IoT Edge devices and modules (edgeAgent/edgeHub) in production, including provisioning, patching, updates, and day-to-day troubleshooting.
- Plan and execute quarterly Azure IoT Edge upgrade and maintenance activities -- upgrading the IoT Edge runtime across the fleet within scheduled maintenance windows, with minimal disruption to store/edge operations.
- Perform L3-level Linux system administration: user/access management, kernel and OS-level troubleshooting, performance/capacity tuning, and root-cause analysis for recurring or high-severity incidents.
- Operate and maintain the observability stack (OpenObserve for logs, PrometheGrafana for metrics and dashboards) across edge, application, and cloud systems.
- Support disaster recovery execution between primary edge/on-prem infrastructure and cloud-based DR environments, including running failover/failback procedures and DR drills.
- Build, maintain, and troubleshoot CI/CD pipelines for application and infrastructure deployment.
- Write and maintain automation scripts/playbooks (Ansible/Puppet/Chef, shell, Python) to reduce manual operational work.
- Respond to alerts, triage incidents, and escalate per runbook/on-call procedures; keep runbooks and documentation current.
- Collaborate with cross-functional teams (security, networking, application) on day-to-day operational issues and change requests.
Mandatory Skills
- Strong hands-on, L3-level administration experience on RHE/Ubuntu: advanced troubleshooting and root-cause analysis across kernel, filesystem, storage, and networking issues, plus performance/capacity tuning, security hardening, and user/permission management in production.
- Proven ability to independently diagnose and resolve complex, escalated (L3) production issues -- performing deep root-cause analysis rather than surface-level fixes, and driving permanent remediation for recurring incidents.
- Hands-on experience operating and supporting Azure IoT Edge runtime (edgeAgent/edgeHub), edge modules, and device provisioning/management in production.
- Strong knowledge and hands-on experience running and troubleshooting Docker containers in production, including on resource-constrained edge hardware.
- Working experience with CI/CD pipelines, building and troubleshooting automated build/deploy workflows.
- Mandatory hands-on experience building and maintaining log aggregation, search, and dashboards using OpenObserve for production infrastructure telemetry.
- Mandatory hands-on experience instrumenting metrics pipelines with Prometheus and building actionable dashboards and alerts in Grafana across on-premises and cloud environments.
- Working knowledge of configuration management tools (Ansible/Puppet/Chef) for automating repeatable system administration tasks.
- Proficiency in shell scripting and at least one general-purpose language (Python/Java/C#/Go) for automation and tooling.
- Working knowledge of the full IoT lifecycle: device provisioning, commissioning, remote device management, module/firmware updates, and decommissioning.
Preferred Skills
- Experience migrating from legacy monitoring tools (e.g. Nagios) to modern observability stacks (OpenObserve, PrometheGrafana).
- Exposure to MQTT broker administration in high-throughput IoT environments.
- Experience administering NGINX (reverse proxy/ingress) configuration and tuning.
- Familiarity with certificate lifecycle management (issuance, rotation, expiry monitoring/alerting).
- Experience with CI/CD orchestration platforms.
- Knowledge of edge-computing constraints in distributed/remote environments (limited compute, intermittent connectivity).
- Experience following runbooks and participating in L1/L2/L3 incident escalation models.
- Experience working with cross-functional teams within and outside the organization.
