Pyspark Developer

Full Time
0
Telecommuting not available Travel not required

Job Description

PySpark/ Big Data Integration Developer

The PySpark/Big Data Integration developer is the lead developer role in Data Engineering responsible for development platform of  the data management platform development. Reporting to the head of Data Engineering, this lead will manage a team of developers to build out the integration jobs using Spark and other Big Data/Hadoop framework. This role will be key to the rollout of the  Data platform and will partner with Data Analytics and other technology teams. Our environment is dynamic, fast-paced, and lots of fun.
 
Key Responsibilities:
  • Coordinate with cross-functional teams, testing teams and drive resolution of open items and issues
  • Customer-focused and work well in a team environment
  • Multi-task and work on multiple projects & prioritize correctly
  • Work with cross border technical team members
  • Interface with the business and analytics team
  • Drive test driven development approach across the data environments
  • Develop using a CI/CD framework setting up best practises where necessary
 
Qualifications:
  • 8 + years experience in Data Warehousing in Media industry and consumer data
  • 3 + year of technology  experience of onshore and offshore reporting development team
  • 3 + years of Agile development
  • 3 + years Experience with Hadoop Ecosystem including Spark, Storm, HDFS, Hive, HBase and other NoSQL databases
  • BS in Computer Science or similar technical degree
  • Experience in developing Spark Streaming applications analyzing the data through Spark (conducted ETL processes and connected to different SQL and Redshift databases)
  • Experience in writing queries for moving data from HDFS to Hive and analyzing data
  • Understanding Partitions, Hive Query optimization, Bucketing etc.
  • Experience in Sqoop for moving data between RDBMS and HDFS
  • Strong understanding of programming paradigms such as distributed architectures and multi-threaded program design
  • Should be very strong in algorithms, collection framework and build high performance engine to handle large amount of data
  • Experience working in a Data warehouse environment and ETL is a huge plus
 
Dice Id : esi
Position Id : 480-1
Have a Job? Post it

Similar Positions

Senior Scala Engineer
  • BizTek Innovations
  • Stamford, CT
Cognos developer(With pharmaceutical experience)
  • Global Talent Resources
  • Piscataway, NJ
Talend Big Data with AWS
  • Idexcel Inc.
  • Stamford, CT
ETL Talend Architect, Remote
  • Stellent IT LLC
  • Long Island City, NY
F2F Interview Must - Senior Business Analyst_ Melville, NY
  • Svam International, Inc.
  • Melville, NY
Scala/Java/Cloud/Big Data Engineers
  • Case Interactive
  • New York, NY
Hadoop Pyspark Developer
  • Corporate Biz Solutions Inc
  • New York City, NY
BigData Developer
  • Top Source International Inc.
  • New York, NY
END CLIENT l Sr. Hadoop/LINUX System Admn
  • Ordusion Technologies, Inc
  • New York City, NY
Spark Developer at NYC,NY
  • Saksoft
  • New York City, NY
Hadoop/Solr Developer
  • Spyglass Partners LLC
  • New York, NY
Cassandra Developer/Architect
  • Atlantic Partners
  • New York, NY