Data Engineer

python, aws, airflow, sqoop, hadoop, snowflake, data engineer, java
Full Time
Depends on Experience
Work from home not available Travel not required

Job Description

Role responsibilities:

  • Design and implement features in collaboration with team engineers, product owners, data analysts, and business partners using Agile / Scrum methodology

  • Contribute to overall architecture, frameworks and patterns for processing and storing large data volumes

  • Design and implement distributed data processing pipelines using Spark, Hive, Sqoop, Python, and other tools and languages prevalent in the Hadoop ecosystem

  • Build utilities, user defined functions, and frameworks to better enable data flow patterns

  • Research, evaluate and utilize new technologies/tools/frameworks centered around high-volume data processing

  • Define and apply appropriate data acquisition and consumption strategies for given technical scenarios

  • Build and incorporate automated unit tests and participate in integration testing efforts

  • Work with architecture/engineering leads and other teams to ensure quality solutions are implemented, and engineering best practices are defined and adhered to

  • Work across teams to resolve operational and performance issues

 

Requirements:

  • MS/BS in Computer Science, or related technical discipline

  • 5+ years of industry experience, 3+ years of relevant big data/relational db experience

  • 5 + years in Unix systems engineering with experience in Red Hat Linux, Centos or Ubuntu 

  • Troubleshooting production issues and performing On-Call duties, at times.

  • Experience with AWS Virtual Private Cloud, EC2, AWS Data Pipeline, AWS Cloud Formation, Auto Scaling, AWS Simple Storage Service, EMR or other AWS products.

  • 4+ year experience in Python

  • Snowflake and Airflow are very nice to have

  • Experience working with Hadoop and related processing frameworks such as Spark, Hive, Sqoop, is a very nice to have

  • Experience with RDBMS systems, SQL and SQL Analytical functions


Posted By

Thorpe Thorpe

506 SW Sixth Avenue #1000, Wilcox Building Portland, OR, 97204

Contact
Dice Id : RTX169535
Position Id : 5865961
Have a Job? Post it