Job Title: Data Engineer/Hadoop Engineer
Primary Skills: data pipelines,spark,scala/kafka
Location: Sunnyvale, CA
Duration : 18+ months
Role and responsibilities : Build big data event driven applications Kafka consumer Build / Scale / Debug - Realtime stream processing framework with Apache Flink / Spark Build new / debug DAG operators in APEX / Airflow Apache Druid - event driven stacks with Kafka and data lakes (hdfs) : Build new/ debug existing Experience in programming in Scala / Python / Java In depth understanding of the Hadoop infrastructure setup/ configuration Spark / Hive SQL Oozie workflow (Or knowledge and experience of other scheduler) - written/ maintained and managed Strong ETL experience
- Not just extraction from data lakes and loading into HDFS.
- Experience with transformations in spark. (DAG) Additional criteria : Engineer / dev-ops experience with Kafka consumer or similar technology Experience with relational databases
- data model / SQL Coding - Algorithms / Pseudo and actual coding concepts (OOP / hash etc.)
Thanks & Regards
Shaik Sadeq (Sam)
Infobahn Soft world Inc.