Data Engineer with Spark, performance tuning

Spark, Performance Tuning, (MySQL OR PostgresSQL OR SQLServer), Python
Full Time
Depends on Experience
Travel not required

Job Description

This position is 100% remote. 

Fulltime position with benefits (Medical, dental, vision), holidays and vacation. 

Position Details: 

As a Data Engineer, you will be a part of an early-stage team that builds the data transport, collection, and storage, We are looking for a Data Engineer to build a scalable data platform. You'll have ownership of our core data pipeline that powers client’s top line metrics; You will also use data expertise to help evolve data models in several components of the data stack; You will help architect, building, and launching scalable data pipelines to support growing data processing and analytics needs. Your efforts will allow access to business and user behavior insights, using huge amounts of data to fuel several teams such as Analytics, Data Science, Marketplace and many others.

Responsibilities:

Owner of the core company data pipeline, responsible for scaling up data processing flow to meet the rapid data growth.

Evolve data model and data schema based on business and engineering needs

Implement systems tracking data quality and consistency

Develop tools supporting self-service data pipeline management (ETL)

SQL and MapReduce job tuning to improve data processing performance

Experience:

5+ years of relevant professional experience

Experience with Hadoop (or similar) Ecosystem (MapReduce, Yarn, HDFS, Hive, Spark, Presto, Pig, HBase, Parquet)

Proficient in at least one of the SQL languages (MySQL, PostgreSQL, SqlServer, Oracle)

Good understanding of SQL Engine and able to conduct advanced performance tuning

Strong skills in scripting language (Python, Ruby, Bash)

1+ years of experience with workflow management tools (Airflow, Oozie, Azkaba

Dice Id : SIGMACON
Position Id : 7514035
Originally Posted : 4 weeks ago
Have a Job? Post it

Similar Positions

Urgent requirement - Spark/Scala Data Engineer - Please respond.
  • Precision Technologies Corp
  • Cat Spring, TX, USA
Big Data Engineer Hadoop, Hive, Scala, Python, Spark
  • KPI Partners, Inc.
  • Dallas, TX, USA
Spark/Scala Data Engineer
  • Judge Group, Inc.
  • Austin, TX, USA
Spark Data Engineer- Full Time
  • Computomic
  • Princeton, NJ, USA
Big Data (Hadoop/Spark) Support Engineer
  • Net2Source Inc.
  • Bellevue, WA, USA
Senior Data Engineer (Spark, Python, SQL)
  • Motion Recruitment
  • Irvine, CA, USA
Python Data Engineer (with Snowflake and Spark)
  • Motion Recruitment
  • San Francisco, CA, USA
Data Engineer (Python, Redshift, Spark, Kafka)
  • Motion Recruitment
  • Irvine, CA, USA