Technical Lead – Data Engineering

Position Overview

We are seeking an experienced Technical Lead – Data Engineering to lead a team of 5–6 data engineers and drive the design, development, and implementation of scalable, high-performance data engineering solutions.

The ideal candidate will possess strong hands-on expertise in Python, Apache Airflow, Snowflake, Trino, Hive, Apache Kafka, and SQL, along with experience in data pipeline development, workflow orchestration, and distributed data processing. This role requires a balance of technical leadership, hands-on engineering, performance optimization, and team mentoring.

Key Responsibilities

• Lead, mentor, and guide a team of 5–6 data engineers, fostering technical excellence, collaboration, and continuous improvement.

• Design, develop, and maintain scalable, reliable, and efficient data pipelines using Python and Apache Airflow.

• Develop and manage Airflow DAGs for workflow orchestration, scheduling, monitoring, and dependency management. Experience with Apache Airflow 3 is preferred.

• Design, implement, and optimize data processing solutions using Snowflake, Trino, and Apache Hive.

• Work with Snowflake architecture, including data storage, compute, data loading, and query performance optimization.

• Develop and optimize complex SQL queries across data platforms, ensuring efficient data retrieval and processing.

• Leverage Trino for distributed SQL query execution and data access across multiple data sources.

• Utilize Hive for large-scale data processing, querying, and management of data stored in distributed environments.

• Build and maintain real-time streaming and event-driven data pipelines using Apache Kafka and Spark Streaming.

• Develop Python-based data transformation, integration, and automation solutions.

• Build and integrate REST/SOAP APIs to facilitate data exchange across multiple systems and platforms.

• Drive technical design decisions, establish coding standards, and implement data engineering best practices.

• Conduct code reviews, troubleshoot complex technical issues, and ensure data quality, reliability, and performance.

• Collaborate with cross-functional teams, business stakeholders, and architects to translate requirements into effective technical solutions.

• Follow Agile/Scrum methodologies and ensure timely delivery of high-quality engineering solutions.

Required Qualifications

• 7–8+ years of hands-on experience in data engineering, data platform development, or related fields.

• 3+ years of technical leadership experience preferred, including mentoring engineers and guiding technical delivery.

• Strong programming skills in Python, including data processing, scripting, automation, and pipeline development.

• Hands-on experience with Apache Airflow, including DAG development, workflow orchestration, scheduling, and monitoring. Airflow 3 experience is preferred.

• Strong understanding of Snowflake architecture, data warehousing concepts, and query optimization.

• Practical experience with Trino, including distributed SQL querying and data access across multiple data sources.

• Experience with Apache Hive, including HiveQL, data querying, and large-scale data processing.

• Advanced SQL skills, including complex queries, joins, analytical functions, and performance tuning.

• Hands-on experience with Apache Kafka and event-driven data architectures.

• Experience with Spark Streaming or similar real-time data processing technologies.

• Knowledge of REST/SOAP API development, integration, and data exchange across systems.

• Understanding of scalable data pipeline design, data integration, data transformation, and data quality practices.

• Strong problem-solving, analytical, communication, and stakeholder management skills.

• Experience working in Agile/Scrum environments.

Preferred Qualifications

• Experience with Apache Airflow 3.

• Experience designing and optimizing distributed data processing and query execution workflows.

• Exposure to cloud-based data platforms and modern data engineering architectures.

• Experience establishing engineering standards, improving pipeline performance, and mentoring technical teams.

What We Offer

• Opportunity to lead impactful data engineering initiatives and modern data platform development.

• A collaborative and innovative work environment.

• Opportunities to drive technical strategy, mentor engineering talent, and implement scalable data solutions.