Big Data Engineer

Big data, Spark, hdfs, Hadoop, SQL, MPP, Erwi, Visio, Python, Java, Data Modeling, Kafka, Cloud computing, machine learning, text analysis, NLP, HBase, Cassandra, NoSQL, Equities, Options trading, finan
Full Time
Negotiable
Work from home not available Travel not required

Job Description

What you will be doing?

Build massive reservoirs for Big data

Design, Develop, construct, test and maintain architectures such as large-scale data processing systems

Tool selection and POC analysis

Gather and process raw data at scale that meet functional / non-functional business requirements (including writing scripts, REST API calls, SQL Queries, etc.)

Develop data set processes for data modeling, mining and production

Integrate new data management technologies and software engineering tools into existing structures

Create custom software components (e.g. specialized UDFs) and analytics applications

Employ a variety of languages and tools (e.g. scripting languages) to marry systems together

Install and update disaster recovery procedures

Recommend ways to improve data reliability, efficiency and quality

Collaborate with data architects, modelers and IT team members on project goals

Support our business users with ad-hoc analysis and reports

Build high-performance algorithms, prototypes, predictive models and proof of concepts

Research opportunities for data acquisition and new uses for existing data

What you bring to the table:

Knowledge and Experience

Master’s degree in Computer Science, software/computer Engineering, applied math, physic, statistics or related field (or relevant work experience and technical expertise)

Hadoop based technologies (e.g. hdfs, Spark). Spark Experience is must

Strong SQL skills on multiple platform (preferred MPP systems)

Database Architectures

Data Modeling tools (e.g. Erwin, Visio)

5+ years of Programming experience in Python, and/or Java

Experience with Continuous integration and deployment

Strong Unix/Linux skills

Experience in petabyte scale data environments and integration of data from multiple diverse sources

Kafka, Cloud computing, machine learning, text analysis, NLP & Web development experience is a plus

NoSQL experience a plus (HBase, Cassandra)

Finance experience, most notably in Equities and Options trading and reference data is a plus

Please send me a copy of your resume if you are open to learning more.

Due to the large volume of applications we receive, we will only be able to contact selected candidates at this time. 

Dice Id : hyattldr
Position Id : JLQAAnalyst
Have a Job? Post it

Similar Positions

JUNIOR Business Analyst
  • Alpha Technologies Inc
  • New York, NY
Business Analyst
  • Optima Global Solutions
  • New York City, NY
IT Business Analyst/PM
  • AH Infotech
  • New York, NY
business analyst - HBO - data analytics engineering projects
  • The Randy Neuringer Company
  • New York, NY
Junior BA
  • APT USA Inc.
  • New York, NY
Sr Business Analyst
  • Impetus Systems and Careers
  • Brooklyn, NY
Windows Security Engineer : NYC
  • New York Technology Partners
  • New York, NY
Sr System Analyst(459-1)
  • Cardinal Integrated Technologies Inc
  • Union, NJ
Sr Business Analyst
  • Useready
  • Union, NJ
Business Analyst
  • Trigyn Technologies, Inc.
  • New York City, NY
Sr. System Analyst
  • Qstride
  • Newark, NJ
Murex Business Analyst
  • ConsultNet, LLC
  • New York, NY