Senior Ansible / AAP Automation Developer, RHEL LEAPP
Remote
Long Term Contract
Summary:
Seeking an experienced Ansible Automation Platform (AAP) developer with strong Red Hat Enterprise Linux administration skills to support and continually enhance automated RHEL in-place upgrades using Red Hat LEAPP.
This is a sustaining engineering role responsible for improving the reliability, scalability, and functionality of the upgrade automation, with the objective of driving upgrades toward 100% successful completion without manual intervention.
The engineer must be capable of troubleshooting failures across both the Linux operating system and automation code, identifying recurring failure patterns, and developing permanent automated remediation rather than relying on operational workarounds.
Core Responsibilities
Develop, maintain, troubleshoot, and enhance Ansible/AAP automation for RHEL LEAPP in-place upgrades.
Continuously improve upgrade success rates with a goal of fully automated, zero-touch upgrades.
Analyze failed or partially completed upgrades and determine the underlying technical cause.
Convert recurring manual remediation steps into Ansible automation and reusable roles.
Develop automated:
o Pre-upgrade readiness checks
o LEAPP preupgrade analysis
o Inhibitor detection and remediation
o Application/service validation
o Repository and package validation
o Disk-space and filesystem checks
o Kernel and boot configuration validation
o Post-upgrade validation
o Failure recovery and rerun capabilities
Improve automation resilience, including error handling, retry logic, idempotency, logging, exception handling, and recovery from partially completed workflows.
Develop and maintain Python utilities/modules where Ansible alone is insufficient.
Maintain automation through Git-based source control and Azure DevOps (ADO).
Develop automated testing and regression capabilities to ensure changes do not introduce failures in previously successful upgrade scenarios.
Support production upgrade events and perform root-cause analysis of automation and RHEL/LEAPP failures.
Work closely with Linux Engineering to incorporate new scenarios, exceptions, application requirements, and RHEL changes into the automation.
Required Technical Skills
Red Hat / UNIX Engineering
Strong hands-on RHEL/Linux system administration experience.
Deep understanding of:
o RHEL 7, 8, and 9
o RPM/YUM/DNF package management
o Systemd
o Kernel and boot processes
o GRUB
o LVM and filesystems
o Multipathing/storage
o Networking
o Authentication and PAM
o SELinux
o SSH
o Repository management
o System logging and troubleshooting
Experience troubleshooting OS upgrade, package dependency, boot, repository, filesystem, and configuration issues.
Experience with Red Hat LEAPP or RHEL in-place upgrades strongly preferred.
Ansible / Automation
Advanced Ansible development skills.
Strong experience with Red Hat Ansible Automation Platform (AAP).
Ability to develop and maintain:
o Playbooks
o Roles
o Collections
o Inventories
o Variables and facts
o Jinja2 templates
o Handlers
o Custom modules/plugins
o AAP job templates and workflows
Strong understanding of idempotent automation design.
Experience designing workflows that can safely recover from failures and resume execution.
Experience with Ansible error handling using constructs such as block, rescue, conditional execution, validation, retries, and structured failure reporting.
Development
Strong Python development and troubleshooting skills.
Shell scripting, particularly Bash.
Experience consuming and integrating REST APIs.
Ability to parse and normalize structured and unstructured system output using JSON, YAML, regex, and similar techniques.
DevOps / SDLC
Strong Git experience, including branching, pull requests, code review, and release/version management.
Experience with Azure DevOps (ADO):
o Repositories
o Pipelines
o Boards/work items
o CI/CD
Familiarity with automated testing of infrastructure code.
Experience maintaining production automation through a controlled software development lifecycle.
Strongly Preferred
Red Hat LEAPP development or operational experience.
Large-scale RHEL lifecycle management experience.
Experience automating upgrades across hundreds or thousands of Linux servers.
Experience integrating AAP with systems such as:
o ServiceNow
o CMDB/inventory platforms
o Scheduling/orchestration systems
o Monitoring platforms
o REST APIs
Experience developing automated health checks and post-change validation.
Experience working in highly controlled enterprise production environments.
Key Engineering Mindset
This is particularly important for the contracting company:
The engineer must approach upgrade failures as automation defects or engineering opportunities, not simply operational incidents.
The expectation is that when an upgrade requires manual intervention, the engineer will:
1. Determine why the automation failed.
2. Identify whether the condition can be detected before the upgrade.
3. Develop automated prevention or remediation.
4. Add the scenario to regression testing.
5. Deploy the improvement through the controlled development process.
6. Measure whether the change improves the overall automated upgrade success rate.
Success Measures
The contractor should be measured against outcomes such as:
Increasing fully automated LEAPP upgrade success rate toward 100%.
Reducing manual intervention during upgrades.
Reducing repeat failures from previously identified conditions.
Increasing the number of automatically remediated LEAPP inhibitors.
Improving automation restart/recovery capability.
Reducing engineering effort required per upgrade.
Maintaining stability as new RHEL/LEAPP updates and application scenarios are introduced.
