Data Engineer/Analyst

Position: Data Engineer/Analyst

Duration: 9 Months

Location: Minneapolis, Minnesota (Hybrid)

The Minnesota Department of Information Technology Services (MNIT), partnering with the Minnesota Department of Labor and Industry (DLI), is seeking one (1) part-time staff augmentation resource to perform Data Engineer/Analyst duties pertaining to the DLI Workers' Compensation division's technology system, Campus.

The Campus system was implemented in November 2020 to support the regulatory role of DLI and the stakeholders across the Workers' Compensation industry.

The system was custom-built from .NET (backend) and Angular (frontend) based on specific technical requirements which resulted in complex code. The system uses Azure-hosted microservice architecture and a complex design framework with minimal design and code documentation.

The resource under this engagement will work in two primary areas: Data Mart and SQL development and AI-driven document classification and disposition.

The Data Mart work involves administering SQL Server databases in Azure, designing and maintaining relational and dimensional data models, and developing stored procedures to support data operations. It also includes performance tuning, documentation of data architecture, and the use of SSRS and SSIS for reporting and integration. A key component is the transfer of knowledge to MNIT and DLI staff to ensure continuity and understanding of the systems and processes in place.

The AI document classification work focuses on identifying and removing redundant or obsolete documents stored in Azure Blob Storage using tools like Azure AI Document Intelligence and automated scripts. This includes developing sampling methodologies, collaborating with stakeholders, and creating classification and retention matrices. The resource will support the development of evaluation tools to measure AI tagging accuracy, maintain audit trails for compliance, and provide training and documentation to state staff. Additionally, the resource will be involved in orchestrating AI processes to minimize system impact and ensure sustainable, legally defensible document management practices.

Data Mart and SQL Development:

• Serve as a Database Administrator (DBA) using SQL Server on Azure to create, update, and drop database tables.
• Analyze and modify various database structures, including relational models and Star Schema design used for the DLI Campus Data Mart.
• Develop and support SQL stored procedures to support Data Mart operations and integrations.
• Update and support the maintenance of documentation (or handover of findings) relating to:
  • Data Mart and relational database architecture
  • Fact tables
  • Dimensional tables
  • Support tables
  • Flow of data into and within the Data Mart
  • Tools and processes for data load and manipulation
  • Setup and maintenance of new and existing data flows
• Execute performance tuning on both databases and Data Mart environments, including providing recommendations for optimization.
• Utilize SSRS (SQL Server Reporting Services) and SSIS (SQL Server Integration Services) tools as needed for reporting and integration.
• Apply data integration best practices to ensure accurate, timely, and secure data ingestion and transformation.
• Develop and support backend operations utilizing SQL.
• Provide knowledge transfer and cross-training to MNIT and DLI staff on data processes, architecture, and toolsets used.
• Perform other related duties as assigned.

AI-Driven Document Classification and Disposition:

• Identify and eliminate redundant, obsolete, or duplicate documents stored in Azure Blob Storage:
  • Use Azure AI Document Intelligence, hashing algorithms, and other programmatic techniques to detect documents that are unlinked to records, redundant, or eligible for deletion, in accordance with applicable retention schedules and policies.
  • Develop a sampling methodology to validate suspect documents, including blank or unnecessary files.
  • Collaborate with business stakeholders to confirm sampling accuracy and disposition direction.
  • Execute systematic removal of validated unneeded documents using scripts or automated methods.
• Collaborate with business stakeholders to define document-related user stories and translate them into actionable AI tagging logic and escalation workflows.
• Develop a Sampling Disposition Matrix and Metadata-to-Decision Matrix guides for classification, review, retention, and deletion paths.
• Develop evaluation tools and logic to measure AI tagging success rates (like Claim ID match 90%, classification 85%, 10% manual review); maintain confidence thresholds and escalation criteria.
• Maintain and support audit trails and traceability features for AI-tagged outputs to enable legal defensibility and compliance tracking.
• Generate reports and/or summary visualizations showing:
  • Tagging progress (processed vs. remaining)
  • Document classification outcomes
  • Match accuracy and confidence scores
  • Flagged documents and reviewer workload
  • Legal and audit risk indicators
• Provide knowledge transfer related to AI model use and document tagging processes:
  • Train assigned State staff in model operation and interpretation
  • Prepare documentation for downstream data tables generated from AI tagging
  • Assist in operationalizing business review workflows for ongoing sampling and validation
• Set up automated processes (utilizing Azure Functions or Data Factories) and schedule AI runs during non-peak hours when needed to minimize performance impact on the system.
• Perform other related duties as assigned.

Desired Qualifications:

Technical Knowledge and Expertise:

• Experience in SQL Server development and administration
• Experience with SQL Server on Azure including experience performing table design, schema evolution, indexing, partitioning, and stored procedure development.
• Experience using T-SQL for ETL routines, business rule implementation, and data quality checks.
• Experience with performance tuning techniques (e.g., query plan analysis, indexing strategies, I/O optimization).

Azure Data Ecosystem

• Experience with core Azure services including Azure Data Factory, Azure Blob Storage, Azure Functions, Azure Synapse Analytics, and Azure Durable Functions.
• Experience building and managing scalable pipelines for batch data ingestion and transformation.
• Experience with Azure AI Document Intelligence for OCR, entity extraction, and classification.

Data Warehousing & Modeling:

• Experience designing and maintaining Star Schema Data Marts, including creation of fact and dimension tables.
• Experience with OLAP/OLTP models and their usage in reporting and analytics environments.
• Experience with metadata documentation, source-to-target mappings, and version-controlled schema evolution.

ETL / Data Integration Tools

• Experience using ETL tools and frameworks including SSIS, ADF, Databricks.
• Experience implementing incremental loads, change data capture (CDC), and transformation logic across varied source systems.
• Experience implementing validation routines and data quality checkpoints within ETL processes.

AI & Automation Capabilities:

• AI-Based Document Tagging and Sampling
• Experience with AI-based document classification and field extraction (e.g., Claim ID, Name, SSN) using Azure Cognitive Services.
• Experience applying hashing algorithms, document similarity checks, and sampling methodologies to detect duplicates or invalid files.
• Experience developing retention logic using metadata-driven disposition matrices aligned to legal and compliance requirements.
• Automation & Job Scheduling
• Experience using PowerShell, Python, and/or Azure-native scripting to automate document cleanup, tagging, and metadata storage.
• Experience using Azure Durable Functions or similar orchestrators to schedule jobs during off-peak hours and ensure process reliability.

Reporting and Visualization:

• Experience with SSRS and Power BI for operational and analytical reporting.
• Experience designing dashboards for tracking project progress and data quality:
• Tagging throughput and confidence scores
• Document classification outcomes
• Legal/audit review volumes
• Data Mart status and load tracking

Soft Skills and Business Alignment:

• Business data analysis and communication
• Experience working closely with legal, business operations, and compliance to define document requirements and metadata use cases.
• Experience gathering business requirements and translating them into technical specifications (e.g., user stories, sampling rules, retention paths).
• Experience creating and maintaining Metadata-to-Decision and Disposition Matrices that align technical tagging with policy decisions.
• Stakeholder Communication and Documentation
• Experience translating policy and retention needs into technical tagging, sampling, and deletion logic.
• Experience synthesizing complex technical findings into non-technical terms for executives and end users.
• Experience leading walkthroughs, facilitating decision points, and presenting findings and recommendations across stakeholder groups.
• Experience incorporating stakeholder feedback and adjusting processes to address issues.
• Experience conducting knowledge transfer and technical training.
• Experience working in Agile environments with cross-functional teams.