Apache Airflow Technical SME

Overview

On Site
Depends on Experience
Contract - Independent
Contract - W2
Contract - 6 Month(s)

Skills

Apache Airflow
Celery
Data Modeling
Continuous Integration
Continuous Delivery
Cloud Computing
ELT
Extract
Transform
Load
Kubernetes
Amazon Web Services
DevOps
Databricks
API Management
Pipeline Management
Microsoft Azure
SQL
Python
Workflow

Job Details

Experience requested:7 to 10 Yrs

Description:
Apache Airflow Technical SME

Strong Python and SQL proficiency, a deep understanding of workflow orchestration and DAG design, and experience with cloud platforms, version control systems (like Git), and monitoring tools. Other essential skills are knowledge of ETL/ELT processes, data modeling, distributed systems, and CI/CD principles.

" Airflow Expertise:
o Possessing deep knowledge of Airflow's architecture, including schedulers, executors (Celery, Kubernetes), and plugin development.
Workflow Design and Development:
o Designing and developing complex, modular, and reusable DAGs (Directed Acyclic Graphs) to automate data pipelines.
Performance Optimization:
o Identifying and addressing performance bottlenecks in Airflow environments and implementing best practices for orchestration and scheduling.
Integration with Cloud Services:
o Integrating Airflow with cloud-native services like Azure Data Factory, Azure Databricks, Azure Storage, and others.
CI/CD Pipeline Management:
o Developing and maintaining CI/CD pipelines for Airflow DAG deployment, testing, and version control, often using tools like Azure DevOps.
Monitoring and Alerting:
o Implementing monitoring, alerting, and logging standards for Airflow jobs to ensure operational excellence and rapid incident response.
Documentation and Knowledge Sharing:
o Creating and maintaining documentation for Airflow configurations, deployment processes, and operational procedures. Mentoring other engineers and leading knowledge-sharing sessions.
Troubleshooting and Incident Response:
o Providing expert-level troubleshooting support for Airflow-related issues and contributing to incident response efforts.
Additional Responsibilities:
Cloud Platform Expertise:
o May require strong knowledge of cloud platforms like Azure (including services like Azure Synapse, Azure Functions, and Azure API Management) or AWS.
Containerization and Orchestration:
o Experience with Docker and Kubernetes for deploying and managing Airflow in containerized environments.
Programming Skills:
o Proficiency in Python/Java for developing Airflow DAGs and other related components. "

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.