Position: Data Engineer with Airflow
Location: USA (Remote)
Justification
We are implementing Astronomer-backed Apache Airflow (Astro Private Cloud) as our enterprise orchestration platform on Azure Kubernetes Service (AKS). This platform provides a centralized, scalable scheduling and workflow orchestration layer for data pipelines and integrations across the organization.
We are seeking a hands-on engineer to support the platform with responsibilities spanning DAG development enablement and platform-level support. The ideal candidate should be comfortable working across Airflow (3.x+), Kubernetes, and cloud infrastructure, while enabling developers to efficiently adopt the platform.
Job Description
Experience - 4 - 6 years
Key Responsibilities
DAG Development & Developer Enablement
Design, develop, and optimize Airflow DAGs for enterprise use cases
Demonstrate strong experience with operators such as Python Operator, SQL Operators, API/HTTP Operators, Kubernetes Pod Operator, etc.
Apply advanced DAG patterns including dynamic DAGs, branching, sensors, retries, and error handling
Troubleshoot DAG failures, performance issues, and integration challenges
Enable orchestration of modern data workflows including DBT pipelines
Provide reusable templates, best practices, and onboarding support to drive self-service adoption Platform Operations & Kubernetes Support
Support Airflow platform deployed on AKS, covering scheduler, webserver/API, workers, and metadata services
Troubleshoot issues related to task execution, scheduling delays, and resource bottlenecks
Diagnose Kubernetes-level issues involving pods, networking, storage, and RBAC
Work with Azure services such as Key Vault, Storage, and networking
Support CI/CD pipelines for DAG and container-based deployments
Improve platform stability, monitoring, and alerting
Required Skills
Strong experience with Apache Airflow (3.x or later) in production environments
Proven experience in DAG development using operators and orchestration patterns
Working knowledge of Kubernetes (AKS preferred) to support Airflow runtime
Expert-level Python knowledge for building, debugging, and maintaining production-grade workflows and supporting libraries
Expertise in writing unit, integration, and DAG validation tests for Airflow workflows, including mocking operators, validating dependencies, and ensuring reliability in CI/CD pipelines
Experience with DBT and modern data pipelines
Experience with CI/CD tools (Git, Jenkins, Docker)
Familiarity with Azure or other cloud platforms
Preferred Skills
Experience with Astronomer/Astro Private Cloud
Exposure to data platforms such as Snowflake, and APIs
Experience with monitoring tools such as Splunk or Prometheus
Knowledge of secrets management using Azure Key Vault
Summary
This role is critical to ensuring reliable platform operations and scalable adoption of the Airflow-based orchestration platform. The ideal candidate combines strong Airflow expertise, Kubernetes knowledge, and hands-on development experience to support both platform stability and developer productivity.
Thanks & Regards
Kundan Mishra
Sr. Technical Recruiter
Never miss an opportunity! Create an alert based on the job you applied for.
