ETL Pipeline/Big Data Expert Pyspark/Python

Overview

Hybrid
Depends on Experience
Accepts corp to corp applications
Contract - Independent
Contract - W2
Contract - 1 Year(s)
No Travel Required
Able to Provide Sponsorship

Skills

Agile
Analytics
Apache Hadoop
Apache Oozie
Big Data
Build Tools
Business Intelligence
Data Security
Data Warehouse
EMC GreenPlum
Data Governance
Distribution
Dimensional Modeling
PySpark
Python
Qlik Sense
MPP
Database Design
Extract
Transform
Load
IBM Cognos Analytics
SQL
Tableau
Warehouse
Workflow
Optimization
Computer Science
VLDB
Database
Data Analysis
TBS
Reporting

Job Details

Description & Requirements

The Risk Engineering team is looking for an experienced engineer to join our growing and diverse group. We build tools that help the company discover, classify, and keep track of data especially sensitive and personal information. Our work helps protect this data, supports privacy regulations like GDPR and CCPA, and makes it easier for teams across the company to understand and use data responsibly. In addition to enabling data protection and privacy, we are also building data inventory and risk analytics capabilities that provide deep visibility into data and support informed, risk-aware decisions. We focus on building reliable systems that bring transparency and trust to how data is managed.


A key objective of this role is for you to help to build and support enterprise level data analytics programs leveraging traditional warehouse technologies, Pyspark, MPP databases and Hadoop.


We'll trust you to:

  • Join a fast-paced team of dedicated engineers who are committed to building risk analytics to drive enterprise-wide data governance and privacy.
  • Shape the strategic and technological direction of the team
  • Empower your career growth through exposure to cutting-edge tools, processes, and challenges
  • Work closely with engineering and risk partners to deliver enterprise-scale impact
  • Translate business needs into robust engineering solutions


In order to be successful:

  • You should have a working knowledge of industry standard Data Infrastructure (e.g. Warehouse, BI, Analytics, Big-Data, etc.) tools with the goal of providing end users with analytics at the speed of thought.
  • You should be proficient at developing, architecting, standardizing and supporting technology platforms using Industry leading ETL solutions.
  • You should thrive in building scalable and high throughput systems
  • You should have experience with agile BI & ETL practices to assist with the interim Data preparation for Data Discovery & self-service needs.
  • You must have strong communication, presentation, problem-solving, and trouble-shooting skills.
  • You should be highly motivated to drive innovations company-wide.


You ll need to have:

  • 5+ years of experience in designing and developing ETL pipelines leveraging pyspark/python.
  • Experience on python database libraries like SQLAlchemy, psycopg2 .etc
  • Strong understanding of data warehousing methodologies, ETL processing and dimensional data modeling.
  • Advanced SQL capabilities are required. Knowledge of database design techniques and experience working with extremely large data volumes is a plus.
  • Demonstrated experience and ability to work with business users to gather requirements and manage scope.
  • Experience in workflow tools such as oozie or Airflow or Tidal
  • Experience working in a big data environment with technologies such as Greenplum, Hadoop and HIVE
  • BA, BS, MS, PhD in Computer Science, Engineering or related technology field


We d love to see:

  • Experience with large database and DW Implementation (20+ TBs)
  • Understanding of VLDB performance aspects, such as table partitioning, sharding, table distribution and optimization techniques
  • Knowledge of reporting tools such as Qlik Sense, Tableau, Cognos

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.