Data Engineer & Insights Analyst

Description:
Diversified Systems is searching for a Data Engineer & Insights Analyst to support the Pennsylvania Longitudinal Data System (PALDS). The selected consultant will transform complex data into trusted, usable information to support program delivery, operational improvements, and executive decision-making across state government agencies. This position will evaluate available data sources, develop analytics products, design and build data pipelines, integrate and transform data from multiple sources, identify and address data quality issues, and establish controls to support responsible use of data.

Responsibilities:

• Create and maintain optimal data pipeline architecture.
• Design and implement cloud-based data solutions using AWS services such as S3, Glue, Redshift, EMR, Athena, Kinesis, and Lambda.
• Build and maintain data orchestration workflows using Apache Airflow, AWS Step Functions, or similar services.
• Assemble large, complex data sets that meet functional and non-functional business requirements.
• Engineer analytical data environments that support advanced statistical analysis, predictive modeling, machine learning, and evidence-building activities.
• Partner with data scientists, research analysts, and policy analysts to ensure enterprise data assets support reproducible research and statistical modeling.
• Identify, design, and implement internal process improvements including automation, data delivery optimization, and infrastructure redesign.
• Build infrastructure for optimal extraction, transformation, and loading of data from multiple data sources using SQL and big data technologies.
• Build analytics tools that utilize data pipelines to provide actionable insights into education and workforce outcomes, operational efficiency, and other key business performance metrics.
• Develop and maintain data pipeline monitoring, alerting, and logging frameworks.
• Implement data quality checks and validation frameworks across pipelines.
• Develop dashboards, reports, analytical models, and other decision-support products.
• Work with Executive, Data, Design, and Support teams to address data-related technical issues and support data infrastructure needs.
• Ensure compliance with Commonwealth policies, data governance standards, privacy requirements, security controls, and interagency data-sharing agreements.
• Create data tools for analytics and data science teams to support the development and optimization of data products.
• Work with data and analytics professionals to improve the functionality of data systems.

Requirements:

• 5 years of hands-on experience with cloud-based data services such as AWS S3, EC2, Glue, EMR, RDS, Redshift, Athena, Lambda, and DynamoDB required.
• 5 years of experience with object-oriented/object function scripting languages such as Python, R, Java, C++, or Scala required.
• 5 years of experience with relational SQL and NoSQL databases including Oracle, Microsoft SQL Server, Postgres, and Cassandra required.
• 5 years of experience with version control systems such as Git and CI/CD pipelines required.
• 5 years of knowledge of data modeling techniques including dimensional modeling, data vault, and star/snowflake schemas required.
• 5 years of familiarity with data cataloging and metadata management tools such as AWS Glue Data Catalog required.
• 5 years of understanding of data governance, data lineage, and data quality frameworks required.
• 5 years of knowledge of REST API design and integration patterns required.
• 5 years of experience with big data tools such as Hadoop, Spark, and Kafka required.
• 5 years of experience with data pipeline and workflow management tools such as Azkaban, Luigi, and Airflow required.
• 5 years of experience with data integration solutions from vendors such as Informatica, MuleSoft, Talend, and TIBCO required.
• 5 years of experience with stream-processing systems such as Storm and Spark Streaming required.
• Advanced SQL knowledge and proficiency working with relational databases and query authoring required.
• Experience building and optimizing big data pipelines, architectures, and data sets required.
• Advanced knowledge and proficiency designing, building, and refining dashboards, reports, analytical models, and decision-support products for executive and non-technical users required.
• Experience working with student-level higher education data and establishing linkages to other sources, including employment and wage data, required.
• Experience performing root-cause analysis on internal and external data and processes required.
• Strong analytical skills working with unstructured datasets required.
• Experience supporting data transformation, data structures, metadata, dependency, and workload management required.
• Experience manipulating, processing, and extracting value from large, disconnected datasets required.
• Working knowledge of message queuing, stream processing, and highly scalable big data stores required.
• Strong project management and organizational skills required.
• Experience supporting and working with cross-functional teams in a dynamic environment required.

About Diversified Systems
Founded in 1990, Diversified Systems is an award-winning Technology Services corporation providing all levels of IT project consulting services nationwide. DSI is headquartered in Columbus, Ohio with regional offices in the American Midwest and East Coast. We offer our consultants a number of flexible and competitive compensation benefit packages.

Diversified Systems is committed to the principles of equal employment. We are committed to complying with all federal, state, and local laws providing equal employment opportunities, and all other employment laws and regulations.

Diversity, Equity and Inclusion
DSI values authenticity and is committed to making sure our employees and partners are valued and respected. At DSI, we believe building a diverse culture is important because we know when people work together, we can achieve better results as a team. DSI realizes that everyone comes from various backgrounds. We celebrate these differences because our employees are our greatest asset as we strive to best meet the needs of those we serve. As part of our ongoing efforts, DSI is focused on advancing equality, diversity, and inclusion by setting high standards to continually evolve our culture. This includes but is not limited to recruiting, community involvement, client delivery, and career development.