Data Engineer - Azure Focus

Overview

Hybrid
Depends on Experience
Full Time

Skills

Azure
Pyspark
C#
Python
Data Engineer
Data
Design
CMMC
Export Control

Job Details

Job Summary

The Data Engineer designs, builds, and maintains scalable data pipelines and architectures that enable efficient data processing and analysis within the institution. This position works closely with data scientists and analysts to ensure data quality, accessibility, and usability, facilitating data-driven decision-making across various departments.

. This is a government initiative that may require access to sensitive data.

Responsibilities

Job Duty 1 -

Design, build, and maintain robust data pipelines to facilitate the efficient collection, storage, and processing of large datasets.

Job Duty 2 -

Implement data integration solutions that combine data from multiple sources to create a unified view for analysis and reporting.

Job Duty 3 -

Optimize data storage and retrieval processes to improve performance and reduce latency in data access.

Job Duty 4 -

Develop and maintain comprehensive documentation of data models, pipelines, and workflows to support knowledge sharing and continuity

Job Duty 5 -

Monitor data systems for performance and reliability, troubleshooting any issues that arise to minimize disruptions.

Job Duty 6 -

Ensure compliance with data governance policies and best practices to protect sensitive information and maintain data integrity.

Job Duty 7 -

Stay current with industry trends and advancements in data engineering technologies to recommend innovative solutions for data management.

Job Duty 8 -

Train and support staff on data tools and technologies to enhance the institution's overall data literacy and analytical capabilities.

Job Duty 9 -

Collaborate with data scientists and analysts to ensure data quality, accessibility, and usability for various analytical tasks.

Job Duty 10 -

Perform other job-related duties as assigned.

Required Qualifications

Educational Requirements

Bachelor's Degree in a related discipline or equivalent combination of education and experience. Advanced certification may be preferred or required.

Required Experience

Four or more years of relevant experience.

Preferred Qualifications

Additional Preferred Qualifications

University experience, application security methodologies, service oriented architecture

Preferred Educational Qualifications

Master's Degree in a related discipline or equivalent.

Preferred Experience

Seven or more years of job-related experience

Higher Education environment, PeopleSoft, Banner, cloud integration technologies, ETL tools such as DataStage, Java development, object-

oriented development, cloud application management, and enterprise system implementation

Preferred Qualifications

2+ years of hands-on experience with Azure Synapse Analytics, including:

Pipelines (Data Integration)

PySpark Notebooks (Data Transformation)

Azure Functions (Orchestration and Automation)

5+ years of experience writing complex SQL queries and working with relational and non-relational databases.

2+ years of experience with one or more programming languages, such as Python, JSON, C#, or Java.

1+ year of experience using Azure DevOps for CI/CD, version control, and release management.

Experience with Microsoft Power Apps for building low-code business applications.

Familiarity with modern data lake architectures, including:

Azure Data Lake and Delta Lake

File formats such as Parquet, CSV, and JSON

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.