Data Engineer

Full Time

  • No Travel Required

Job Description

Location: Plymouth Meeting, PA
Salary: $60.00 USD Hourly - $70.00 USD Hourly
Description: Our client is currently seeking a Data Engineer

The Data Engineer will work with a team of Data Architects, Data Engineers who will facilitate all aspects of company's Enterprise Data. This position will be responsible for transitioning our data to the Cloud utilizing Python, Azure DataBricks, Azure Data Factory, and PostgreSQL. As a Data Engineer you will support the processing, aggregation and reshaping of the data to enable analytics, NLP and ML work conducted by Data Scientists and Specialists.

This job will have the following responsibilities:
  • Use Python (notebooks), Azure Synapse and/or Azure DataBricks to automate ingestion of publicly available information via RSS feeds, scraping, and API calls that are persisted to multiple types and tiers of Azure storage. Python development will require experience utilizing Natural Language Processing (NLP) to validate and relate ingested data.
  • Support the processing, aggregation and reshaping of the data to enable analytics and ML work conducted by Data Scientists and Specialists. Curated and processed data will be distributed by API's to a public facing website as well as used internally by researchers developing reports to highlight pharmaceuticals with high potential to cause a significant impact to one or more areas of healthcare in certain countries. Reports will include healthcare use, infrastructure, service delivery, disease management, patient health outcomes, and healthcare costs. The system will support regular and frequent human curation, review and feedback to ensure all records in the system are up to date and of high quality.
  • Work closely with product development, enterprise architecture, and business experts to continually review, improve, and refine the ingestion of data from harvested trials, news releases & reports, and other sources being captured on a daily basis. This system must be of high quality, excellent performance and optimal costs to operate month to month. Therefore, a thorough understanding of Azure pricing models and which services are best for which use cases is critical.

Qualifications & Requirements:
  • 3-5 years using Python is a requirement for this position.
  • Comprehensive hands-on experience and knowledge of the following Microsoft Azure product areas:
    • Azure Synapse and/or Azure DataBricks
    • Python Notebooks, PySpark and Natural Language Processing (NLP)
    • Azure Blob Storage
  • 5 years hands on experience with enterprise data development, pipelines and relational data
  • Experience with relational SQL such as Microsoft SQL Server and/or PostgreSQL
  • Experience with agile methodologies, especially Kanban, and using tools like Jira and Confluence to facilitate the work
  • Associates/Bachelor's degree in Computer Science or related field preferred. Equivalent professional experience may be considered.


This job and many more are available through The Judge Group. Find us on the web at ;/a>