Data Engineer Lead

Data Engineer Lead

Contract :

Remote

JD

Responsibilities: Collaborate with machine learning practitioners to understand feature and data requirements Work with Engineering teams to collect required data from internal and external systems Develop and maintain production-grade batch and streaming pipelines Setup orchestration and automation of ETL jobs using tools such as Airflow and Jenkins Improve and extend existing data infrastructure services Conduct data exploration, analysis and provide data strategy consultancy Drive and maintain a culture of quality, innovation and experimentation Work in an Agile environment that focuses on collaboration and teamwork Mentor colleagues on best practices and technical concepts of building large scale solutions Qualifications: 8+ years of data engineering experience Experience deploying and running services in AWS and in engineering big-data solutions using technologies like Databricks, EMR, S3, and Spark Experience building streaming pipelines using Kafka, Spark, Flink, or Samza Experience loading and querying cloud-hosted databases such as Redshift and Snowflake Experience designing and developing backend microservices for large scale distributed systems using gRPC or REST Experience with graph-based data workflows such as Apache Airflow, Meson Excellent communication and people engagement skills

In short, we need

streaming data engineer Lead:

Build tooling and low-latency services to enable and support event-driven data pipelines

Expert in spark scala, python, sql

Experience in building large datasets and scalable services

Experience building streaming data solutions using technologies like Databricks, Flink, Spark