Lead, mentor, and guide a team of 5–6 data engineers, fostering technical excellence, collaboration, and continuous improvement.
Design, develop, and maintain scalable, reliable, and efficient data pipelines using Python and Apache Airflow.
Develop and manage Airflow DAGs for workflow orchestration, scheduling, monitoring, and dependency management. Experience with Apache Airflow 3 is preferred.
Design, implement, and optimize data processing solutions using Snowflake, Trino, and Apache Hive.
Work with Snowflake architecture, including data storage, compute, data loading, and query performance optimization.
Develop and optimize complex SQL queries across data platforms, ensuring efficient data retrieval and processing.
Leverage Trino for distributed SQL query execution and data access across multiple data sources.
Utilize Hive for large-scale data processing, querying, and management of data stored in distributed environments.
Build and maintain real-time streaming and event-driven data pipelines using Apache Kafka and Spark Streaming.
Develop Python-based data transformation, integration, and automation solutions.
Build and integrate REST/SOAP APIs to facilitate data exchange across multiple systems and platforms.
Drive technical design decisions, establish coding standards, and implement data engineering best practices.
Conduct code reviews, troubleshoot complex technical issues, and ensure data quality, reliability, and performance.
Collaborate with cross-functional teams, business stakeholders, and architects to translate requirements into effective technical solutions.
Follow Agile/Scrum methodologies and ensure timely delivery of high-quality engineering solutions.
7–8+ years of hands-on experience in data engineering, data platform development, or related fields.
3+ years of technical leadership experience preferred, including mentoring engineers and guiding technical delivery.
Strong programming skills in Python, including data processing, scripting, automation, and pipeline development.
Hands-on experience with Apache Airflow, including DAG development, workflow orchestration, scheduling, and monitoring. Airflow 3 experience is preferred.
Strong understanding of Snowflake architecture, data warehousing concepts, and query optimization.
Practical experience with Trino, including distributed SQL querying and data access across multiple data sources.
Experience with Apache Hive, including HiveQL, data querying, and large-scale data processing.
Advanced SQL skills, including complex queries, joins, analytical functions, and performance tuning.
Hands-on experience with Apache Kafka and event-driven data architectures.
Experience with Spark Streaming or similar real-time data processing technologies.
Knowledge of REST/SOAP API development, integration, and data exchange across systems.
Understanding of scalable data pipeline design, data integration, data transformation, and data quality practices.
Strong problem-solving, analytical, communication, and stakeholder management skills.
Experience working in Agile/Scrum environments.
