Airflow SME

Job Role: Airflow SME

Location: Remote

Note: Airflow experience is mandatory.

Job Description:

Must Have Technical/Functional Skills

• Airflow Expertise: Possessing deep knowledge of Airflow's architecture, including schedulers, executors (Celery, Kubernetes), and plugin development.
• Workflow Design and Development: Designing and developing complex, modular, and reusable DAGs (Directed Acyclic Graphs) to automate data pipelines.
• Performance Optimization: Identifying and addressing performance bottlenecks in Airflow environments and implementing best practices for orchestration and scheduling.
• Integration with Cloud Services: Integrating Airflow with cloud-native services like Azure Data Factory, Azure Databricks, Azure Storage, and others.
• CI/CD Pipeline Management: Developing and maintaining CI/CD pipelines for Airflow DAG deployment, testing, and version control, often using tools like Azure DevOps.
• Monitoring and Alerting: Implementing monitoring, alerting, and logging standards for Airflow jobs to ensure operational excellence and rapid incident response.
• Documentation and Knowledge Sharing: Creating and maintaining documentation for Airflow configurations, deployment processes, and operational procedures. Mentoring other engineers and leading knowledge-sharing sessions.
• Troubleshooting and Incident Response: Providing expert-level troubleshooting support for Airflow-related issues and contributing to incident response efforts.

Additional Responsibilities:

• Cloud Platform Expertise: May require strong knowledge of cloud platforms like Azure (including services like Azure Synapse, Azure Functions, and Azure API Management) or AWS.
• Containerization and Orchestration: Experience with Docker and Kubernetes for deploying and managing Airflow in containerized environments.
• Programming Skills: Proficiency in Python/Java for developing Airflow DAGs and other related components.