Azure Data Engineer

Top Skills - Must Haves

• azure
• engineering
• Data

Top Skills' Details

1. This person should have experience working with python from a true data engineering perspective not application development
2. Director is looking for someone who expertise in any of the following Azure tools/technologies: Azure Synapse, Azure Data Bricks, Azure Data Factory (don't need to be an expert in all 3)
3. Ability to help mentor others and show them the proper way to build out a data lake within Microsoft

Job Description

Primary Role: This position reports to the DHTS-Data Partnerships, Director of Data and Analytics Platforms. This individual will be primarily responsible for the development of data integration and delivery pipelines while also expanding the FHIR-based content stored within Duke s data lake (the Duke Data Hub). These solutions will capitalize on technologies to improve the value of analytical data, improve effectiveness of information stewardship, and streamline the flow of data in the organization. Solutions will focus on using state of the art data and analytics tools including traditional and near real-time integrations, big-data, and delta lake architecture using both extract, load, transform (ELT) toolsets as well as REST APIs and FHIR. The ideal candidate will also be comfortable with data science platforms with proven experience leveraging DevOps and Automation/Orchestration tools.
Essential Tasks/Responsibilities
Create and maintain optimal data pipeline architecture
Develop a data lake on Microsoft Azure using the medallion architecture leveraging a delta lake format for the silver layer
Assemble large, complex data sets that meet functional / non-functional business requirements
Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
Recommend design of analytics solutions which improves data integration, data quality, and data delivery with an eye towards re-useable components
Articulate differences, advantages, and disadvantages between architectural solution methods
Work with Agile team members to document and execute test plans and data validation scripts. Support the code promotion process through development and production as required by using standard CI/CD processes
Develop monitoring, logging, and error notification processes to ensure data is updated as expected and processing metrics reported
Participate in the creation and maintenance of standards for coding, documentation, error handling, error notification, logging, etc.
Accountable for conforming to established architectural, developmental, and operational standards and practices
Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics
Work with stakeholders including the Executive, Product, Data, and Design teams to assist with data-related technical issues and support their data infrastructure needs
Evaluate and recommend development tools
Assist in application and data operations performance tuning
Participate in system architecture design
Create data tools for analytics and data scientist team members that assist them in building and optimizing our product into an innovative industry leader
Work with data and analytics experts to strive for greater functionality in our data systems
Share troubleshooting and maintenance duties

Education: Bachelor's degree in a related field, or four years of equivalent technical experience required

Required Experience: We are looking for a candidate with 5+ years of experience in a Data Engineer role:

Experience implementing data lakes on Microsoft Azure
Experience with relational SQL and NoSQL databases
Experience with data pipeline and workflow management tools such as Azure Data Factory, Synapse Analytics pipelines
Experience with object-oriented/object function scripting languages such as Python or Java.
Experience with Cloud-based analytics platforms such as Azure Synapse Analytics

Required Skills:
Advanced SQL skills and experience working with a variety of relational database management systems
Intermediate to Advance skills in python development
Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement
Strong analytical skills related to working with unstructured datasets
Build processes supporting data transformation, data structures, metadata, dependency and workload management
A successful history of manipulating, processing and extracting value from large disconnected datasets
Working knowledge of message queuing, stream processing, and highly scalable big data data stores
Experience supporting and working with cross-functional teams in a dynamic environment

Desired Skills:
Working experience with the FHIR specification; including implementation.
Knowledge of APIs, API Integration, and API Management
Working knowledge of DevOps & Automation/Orchestration
Knowledge of open-source software solutions and open source as a business model
Technical breadth across application development, enterprise architecture, or application integration
Understanding of Agile methodology