Apache Airflow Engineer - Hybrid

Role - Apache Airflow Engineer

Location - New York, NY (3 Days to Office)

Mode Hybrid

Skills- Cybermation to Apache Airflow migration, Airflow DAG development, shell script/API scheduling, operational validation, runbook creation, and production support

Job Summary

We are seeking Apache Airflow migration resources to support a client initiative focused on migrating existing Cybermation-scheduled scripts to Apache Airflow. The resources will analyze the current Cybermation schedules, understand the existing shell/API invocation patterns, convert the schedules into Airflow DAGs, validate execution behavior, and support production deployment and handover.

The migration objective is to preserve the existing business execution behavior while improving schedule control, monitoring, logging, retry handling, operational visibility, and supportability through Apache Airflow.

Key Responsibilities

• Analyze existing Cybermation job schedules, dependency patterns, runtime parameters, execution frequency, owners, success/failure behavior, and operational support expectations.
• Review existing shell scripts and API/cURL invocation logic to determine the appropriate Airflow DAG design, task structure, error handling, and retry configuration.
• Design, develop, and maintain Apache Airflow DAGs to orchestrate shell scripts, API calls, and dependent jobs with appropriate scheduling and operational controls.
• Configure Airflow schedules, task dependencies, retries, logging, alerting hooks, variables, connections, parameters, and environment-specific configuration as per client standards.
• Perform dry runs, unit testing, schedule validation, API response validation, failure/retry validation, and production-readiness checks for migrated jobs.
• Create migration mapping, validation evidence, deployment notes, operational runbooks, and support handover documentation.
• Coordinate with client SMEs, application owners, infrastructure teams, and release teams to support deployment, issue triage, initial run monitoring, and hypercare.

Preferred Qualifications

• Prior experience migrating jobs from Cybermation or equivalent enterprise schedulers to Apache Airflow.
• Experience deploying or operating Airflow in cloud or containerized environments is preferred.
• Exposure to AWS services such as EKS, S3, CloudWatch, Lambda, Step Functions, IAM, Secrets Manager, or EventBridge is a plus.
• Experience supporting financial services, capital markets, or other regulated enterprise environments is preferred.
• Familiarity with ITSM processes, release governance, change management, and production support procedures.

Education & Experience

• Bachelor's degree or equivalent experience in Computer Science, Information Technology, Engineering, or a related discipline.
• 7+ years of overall engineering experience with hands-on Airflow, Python, and Linux/shell exposure.
• Strong practical understanding of job scheduling, batch/workflow orchestration, production readiness, and operational support processes.

Role Deliverables

• Migration inventory and mapping of Cybermation schedules to target Airflow DAGs.
• Developed and configured Airflow DAGs for in-scope shell script and API/cURL-based workflows.
• Validated execution evidence covering dry runs, schedule validation, API response checks, failure/retry validation, and defect remediation.
• Deployment notes, operational runbooks, manual re-run steps, monitoring instructions, and production support handover documentation.
• Post-deployment validation and hypercare support for migrated workflows.

Tools & Technologies

Apache Airflow, Python, Linux/Unix, Shell Scripting, cURL, REST APIs, Git, CI/CD basics, YAML/JSON configuration, scheduler migration, job monitoring, logging, retry handling, production support, runbook documentation, and optional AWS services such as EKS, S3, CloudWatch, Lambda, IAM, Secrets Manager, EventBridge, and Step Functions.