This role designs, implements and maintains data pipelines for handling large volumes of data in order to scale analytics, machine learning, and other digital technologies. Also creates, manages, and publishes datasets including extracting, transforming, merging and loading data from various data sources.
- Design, create, and maintain scalable data pipelines that ingest and aggregate data from various factories to build out a centralized dataset repository using Microsoft Azure and other third party products.
- Implement processes and solutions that ensure pipeline efficiency and monitor the data quality and pipeline status.
- Collaborate with engineers, data scientists, and other team members to identify and transform data for ingestion, exploration, and modeling.
- Develop data catalogs/data dictionaries.
- Participate in project planning, identifying milestones and deliverables.
- Manage and track activities and efforts.
- Document the design and artifacts that are created as part of the primary responsibilities.
- Learn manufacturing side of business as part of the job.
- Support the current technologies while adhering and following standards and best practices.
- Support/troubleshoot factory data pipelines.
- 5 – 10 years experience in data engineering, cloud engineering, data science or a related field with experience in working with large volumes of data.
- Experience in Microsoft Azure including Azure Data Factory, IoT Edge/IoT Hub, Storage Accounts, or Databricks.
- Python programming experience is a must.
- Experience working with DevOps and/or MLOps.
- Experience with C# and/or Java is desired.
- Manufacturing experience is a plus.
- Microsoft Azure certification is a plus.
- Strong verbal and written communication skills.
- Experience in multi-tasking, excellent attention to detail and be willing and able to learn new technologies quickly.