Lead Data Architect / Data Engineer

We are seeking a hands-on Lead Data Architect/Data Engineer to define architecture and deliver production-grade pipelines across multiple phases of a Databricks platform.

Responsibilities include:

•             Build and maintain automated ingestion and transformation pipelines that bring data from:

o             public portals and open data sources

o             REST APIs

o             internal departmental databases and systems

into Delta tables in Databricks.

•             Partner with the Program Manager to establish and maintain:

o             platform architecture

o             naming conventions and standards

o             technical documentation

•             Collaborate with the Data Hub governance group to support decisions on:

o             platform and data architecture

o             data sharing patterns

o             data contracts and data ownership

o             data quality and reliability standards

•             Design recommendations for protecting sensitive data, including:

o             row-level and column-level access controls

o             separation of confidential/internal data from public datasets

•             Improve reliability through:

o             monitoring and alerting

o             data quality checks and validation frameworks

•             Enable discoverability and usability by publishing datasets and documentation that support plain-language exploration and AI-enabled analysis tools.

•             Write and maintain skill files (instruction files for AI coding assistants) to enforce consistent pipeline patterns and development standards.

•             Support and enable other departments to build pipelines using shared templates and best practices.

Requirements:

•             Strong leadership experience, including executive-level stakeholder engagement

•             7+ years in data engineering, data architecture, or closely related roles

•             Proven experience delivering large-scale, enterprise data engineering / architecture initiatives

•             Strong hands-on expertise in Python and SQL

•             Experience with Apache Spark or cloud data platforms such as:

o             Databricks (strongly preferred)

o             Snowflake, BigQuery, or similar

•             Experience integrating data from REST APIs, databases, and other source systems

•             Excellent communication skills with strong writing and documentation practices

Preferred

•             Hands-on Databricks experience with:

o             Delta Lake

o             Unity Catalog

o             Databricks Jobs

•             Experience governing sensitive/confidential data (access control, masking, privacy requirements)

•             Experience operating within a data governance program (stewardship, data contracts, cross-agency sharing agreements)

•             Experience writing skill files, prompts, or instruction sets for AI coding assistants

•             Experience with GIS / location-based datasets