Sr. Data Engineer

afka, Spark or Python, Hadoop, Graph DB, Dev/Ops, IoT, AirFlow, NiFi, Snowflake, Azure SQL Warehouse), NoSQL, Graph DB in the cloud (AWS, Azure)
Contract W2, 12 Months
Depends on Experience
Work from home not available Travel not required

Job Description


This role will be responsible for engineering end to end data pipeline including ingesting data from a variety of sources such as structured data, social media, sensor data, images, click stream at scale; integrating data using reusable framework and provide curated data for users and applications on the cloud platform. You will work collaboratively, in a consultative way, across the business units, advising on best approaches to support data needs and build data products. You will also work closely with the Data Science team and drive automation of end to end pipeline data science findings into the data stream. If you want to drive innovation and make a difference, we want to talk to you!

There are several mission-critical applications that rely on this data and experience in a highly scalable distributed architecture is essential. Automation of end to end data pipeline with metadata, data quality check and audit is core to the architecture.
Design, build and test end to end data pipeline including data ingestion (streaming, events and batch), data integration, data curation (on MPP, Graph db)
Build and support the Big data platform on the cloud
Define and implement automation of jobs and testing
Optimize the data pipeline to support ML workloads and use cases
Support mission-critical applications and near real-time data needs from the data platform
Build and support reusable framework to ingest, integration and provision data
Capture and publish metadata and new data to subscribed users
Work collaboratively with product managers, data scientists as well as business partners and actively participate in a design thinking session
Participate in design and code reviews
Motivate, coach, and serve as a role model and mentor for other development team associates/members that leverage the platform

Required Skills:
7+ years experience in data warehouse/data lake technical architecture
Minimum 3 years of Big Data and Big Data tools in one or more of the following: Kafka, Spark or Python, Hadoop
2-3 years experience with MPP database (Snowflake, Azure SQL Warehouse), NoSQL, Graph DB in the cloud (AWS, Azure)
Strong familiarity with batch processing and workflow tools such as AirFlow, NiFi
Experience building distributed architecture and IoT in a Dev/Ops environment
Ability to work independently with business partners and management to understand their needs and exceed expectations in delivering tools/solutions
Strong interpersonal, verbal and written communication skills and ability to present complex technical/analytical concepts to an executive audience
Strong business mindset with customer obsession; ability to collaborate with business partners to identify needs and opportunities for improved data management and delivery
BS/BA degree in Computer Science, Physics, Mathematics, Statistics or other Engineering disciplines
Preferred Skills:
Masters in Computer Science, Physics, Mathematics, Statistics or other Engineering disciplines
Excellent planning, prioritization, problem-solving skills
Experience with data visualization tools such as Looker, PowerBI
Experience with Hortonworks Hadoop implementation

Dice Id : compexpr
Position Id : 1994
Have a Job? Post it

Similar Positions

Senior Data Analyst
  • Certec Consulting Inc
  • Boston, MA
Data Engineer - Boston, MA
  • UnitedHealth Group
  • Boston, MA
Data Engineer
  • CyberCoders
  • Cambridge, MA
Sr.Data Analyst
  • Mass Mutual Financial Group
  • Boston, MA
Data Science
  • Wipro Ltd.
  • Marlborough, MA
Lead Data Engineer
  • Gardner Resources Consulting, LLC
  • Boston, MA
Data Engineer
  • Eliassen Group
  • Weston, MA
Data Engineer
  • The Ceres Group
  • Boston, MA
Data Engineer
  • Robert Half Technology
  • Boston, MA
Data Engineer
  • Putnam Investments
  • Boston, MA
Data Scientist
  • Cubic Technologies
  • Boston, Massachusetts
Data Analyst
  • Hollister
  • Cambridge, MA