Job Title: Azure Senior Data Engineer (Data Catalog)
Client: Oil and Gas
Direct Hiring (No C2C or C2H)
Location: Houston, TX (Onsite - 3 days Remote 2 days)
Strong Data Cataloging and Data Pipeline Experience
Python, Java or Scala, Azure, Azure Cloud or Other Cloud Platforms, Informatica or others listed.
The Data Engineer will be responsible for data governance, data management, data security, and working with complex datasets to support a Data Product development roadmap.
· Experience designing and implementing data catalogs
· Involvement in the scanning and ingestion of data sources and assets and metadata mapping
· Application of data catalog collections, custom classifications, user groups, etc.
· Provide subject matter expertise and hands on delivery of data capture, curation and consumption pipelines on Azure
· Delivery of design/architecture for transformations and modernizations of enterprise data solutions using Azure cloud data technologies.
· Design and Build Modern Data Pipelines and Data Streams.
· Design and Build Data Service APIs.
· Design and optimizing data models on Azure cloud using Azure data stores
· Experience integrating Azure security services with Azure data services for building secure data solutions.
· Architecting and operating large production Hadoop/NoSQL clusters on premise or using Cloud services.
· Expose data to end users using Power BI, Azure API Apps or other modern visualization platform or experience.
· Bachelor’s degree in Computer Science, Information Systems, Business Administration or a related discipline
· 10+ years hands-on experience in architecting, developing, and successfully operationalizing complex/large scale data management projects
· At least 3-5 years’ experience in Python, and ideally also in Java, SQL, or Scala
· At least 3-5 years of delivery experience on Azure
· At least 3-5 years of experience in MS PaaS data services, but not limited to, such as Azure data factory, Azure data lake store, Azure function apps, Azure SQL/ Data Warehouse.
· At least 3-5 years of experience in developing, designing, and deploying end to end batch / real-time solutions.
· Professional experience in developing optimized stored procedures and performing analytical functions on the data sets.
· Experienced in enterprise ETL tools like Data Services, SSIS, Informatica or likewise.
· Previously worked in agile and scrum
· Experience with data cataloging tools such as Alation, Collibra, or Informatica
· DevOps on an Azure platform
· Real-time ingestion: Kafka, Flume, etc.
· IoT, event-driven, microservices, containers/Kubernetes in the cloud
· MCSA Cloud Platform (Azure) Training & Certification
· MCSE Cloud Platform & Infrastructure Training & Certification
· Understanding of statistics and mathematical techniques to solve real business problems.
· Experience of working with models in SAP HANA, manufacturing historians, various IoT, subscription and public data sources.
· Experience of Reporting and Analytics tools such as Tableau, Power BI, and Alteryx
· Functional experience in one of more business functions.
· Experience with global enterprise environment or major consulting firm is a plus.