Key Responsibilities
• Architect and implement large-scale data orchestration frameworks using Apache Airflow.
• Design, develop, and optimize ETL/ELT pipelines for batch and near real-time processing.
• Build scalable workflows integrating Hadoop, Spark, Kafka, Databricks, data lakes, and data warehouses.
• Define Airflow deployment strategies, DAG standards, security, monitoring, and governance.
• Implement workflow scheduling, dependency management, error handling, retries, alerting, and SLA monitoring.
• Lead cloud-based data platform implementations on AWS
• Optimize pipeline performance, scalability, reliability, and cost efficiency.
• Establish CI/CD processes and DevOps practices for Airflow environments.
• Collaborate with business stakeholders, data engineers, architects, and platform teams to translate requirements into technical solutions.
• Mentor engineering teams on Airflow architecture, data orchestration patterns, and best practices.
Required Skills
Airflow & Data Engineering
• Expert knowledge of Apache Airflow DAG design, scheduling, operators, sensors, and executors.
• Strong experience in workflow orchestration, metadata management, and pipeline monitoring.
• Hands-on expertise in ETL/ELT development and data pipeline automation.
Big Data Technologies
• Apache Spark / PySpark
• Hadoop ecosystem
• Apache Kafka
• Data Warehousing concepts
• Data Lakes and Lakehouse architectures
• NoSQL databases
Programming
• Python (advanced)
• SQL (advanced)
• Shell scripting
• Core Java (preferred)
Cloud Platforms
• AWS (EMR, Glue, S3, Redshift)
Preferred Qualifications
• 10+ years of experience in data engineering and big data platforms.
• 3+ years of architecture experience designing enterprise data solutions.
• Experience with Kubernetes, Docker, and cloud-native deployments.
• Knowledge of Data Mesh, Data Fabric, and Modern Data Platforms.
• Experience leading architecture reviews and technology strategy discussions.
Success Metrics
• Pipeline reliability and SLA adherence
• Workflow performance optimization
• Reduced operational incidents
• Improved data quality and observability
• Scalable and reusable orchestration frameworks
.jpg?format=webp)
