Azure Data Engineer Architect

Role: Azure Data Engineer Architect

Location: Brentwood, TN

Description

Lead Azure Data Engineer Architect role is to strategically design, develop and implement Azure Cloud based Data Warehouse, Data Lake, Data Lakehouse, Data Marts & BI solution using modern data engineering techniques and advanced analytics methods to provide information & insights to end users. The role requires modeling, collecting, harmonizing, aggregating, storing and reconciling data from variety of data sources for our petroleum refineries IT/OT technology platform including time series Database, SCADA data, batch, real time, enterprise applications, Rest API's , streaming & IOT data. The role will help design and build data pipelines, streams, data warehouse, data lake, reporting application, data generators and a whole range of tools to provide information and insight.

The ideal candidate will be responsible for designing, developing, and maintaining scalable data processing pipelines in a cloud-based environment. The candidate should have a deep understanding of Azure ADF, Databricks, streaming analytics, data engineering best practices, and a proven track record of delivering high-quality real time solutions.

• Minimum of 12 to 15 years of Data Engineering, Data Warehouse/ Business intelligence with hands-on experience leading and delivering multiple full life cycle implementations.
• Comprehensive data warehouse analysis and design experience, with full knowledge of data warehouse methodologies and data modeling.
• Hands-on experience & deep understanding in designing, development and implementation of Azure Data & Analytics architecture, data modeling, transformations, and development.
• In-depth knowledge and minimum 5 to 7 years' experience developing & implementing data engineering solutions using Azure services for Data Warehouse, Data Lakehouse, & Advance analytics- Primarily- Azure Data Factory, ADLS, Azure Databricks, Azure Functions, Logic Apps, IoT Hub, Event Hub & Azure SQL
• Experience with Data pipeline and workflow management tools & Azure DevOps processes.
• Strong proficiency in programming languages such as Python, PySpark, or Scala
• End to End Implementations expertise in Azure Data Warehouse & Data Lakehouse
• Solid understanding of database structure principles, Master and Reference Data management
• Experience & expertise in optimizing the ETL/ELT data pipeline technology options to minimize the on-going operational cost of Azure data engineering & storage solutions.
• 8+ years experience in agile Data warehouse development methodology.
• 8+ years' experience designing and developing dimensional models for data warehousing
• 3+ years' experience in MSBI suite primarily SSIS
• 3+ years of implementing operational data model is plus not essential
• 3+ years of experience with big data batch/streaming processing.
• 5+ years' experience using Pyspark, Python or Scala
• Experience delivering reporting solution using visualization tools like PowerBI
• Experience in implementing SAP data analytics solution in Azure is plus not essential
• Experience using data modeling tool like ERWin
• Ability to lead and work with remote & offshore teams.
• Experience integrating & analyzing data from time-series databases, SCADA data, IOT, voice to text, images & videos for creating Industrial Intelligence solution.
• Knowledge/expertise on Azure ML/AI predictive modeling capabilities is a plus
• Experience using proven methods to solve business problems using Azure Data and Analytics services in combination with building data pipelines, data streams and system integration
• Hands-on experience with analytics and big data technologies within Microsoft Azure Cloud platform in tools such as Azure Machine Learning & Azure Cognitive Services is plus not essential
• Experience with data profiling, cataloguing, and data mapping for technical design using a use case-based approach that drives the construction of technical data flows
• Experience with Azure Administration & Security (Connection setup, User setup, Object security) is plus
• Understanding of statistical and machine learning modeling with experience applying these modeling techniques to business problems is a plus not essential.
• Ability to set up data and experimental platforms using Azure cloud service
• Ability to work independently with business stakeholders to collect and understand requirements for reporting & analytics.
• Full project management and software development life cycle experience.
• Experience with data process flowcharting techniques.
• Oil & Gas industry experience preferred but not essential.
• Experience & knowledge of SAP S4 and SAP BW4HANA is a plus not essential.
• Excellent quantitative, analytical & mathematical skills is a plus
• Strong understanding of the organization's goals and objectives

Formal Education & Certification

• Bachelor's degree in computer science, information systems, or a related field, or equivalent work experience.
• Certifications in Azure Data Engineering and Analytics space is a plus