Lead Data Engineer

About the Role:

Wilcore is hiring a Data Engineer to support an enterprise cloud-based data and analytics platform for a federal health agency. The platform provides governed data storage, engineering and analytics tools, platform integrations, and policy controls for data practitioners across the enterprise, and is modernizing the tooling available to users of data currently held in a legacy on-premises data warehouse.

In this role you will deliver reliable, scalable ingestion and shared-service data engineering for new and migrated sources, enforcing data quality as part of the definition of done so that datasets are accurate, fresh, and modeled for analytics, reporting, and data science.

Contract Contingency Notice:This position is contingent upon contract award. Employment offers will be extended only upon successful contract award and client approval. Candidates may be considered and interviewed in advance to support rapid onboarding should the contract be awarded.

What you'll be doing:

• Ingest New and Migrated Sources: Ingest new and migrated data sources into the platform using fit-for-purpose patterns, selected per source characteristics, operational constraints, and approved architecture decisions.
• Provide Shared-Service Data Engineering: Transform ingested data into consumption-ready models, including semantic views, conformed dimensions, and performance-oriented partitioning for analytics, reporting, and machine learning.
• Operate Delta Lake Patterns and Dedicated Pool: Operate and improve the platform's delta lake patterns and dedicated pool usage, ensuring parity of data and models migrated from other platforms before customer release.
• Implement Definition-of-Done Quality Checks: Implement quality checks for each domain: equivalence and parity tests against systems of record and, where applicable, prior platforms; freshness SLAs and anomaly detection tied to pipeline events, publishing pass and fail results and residual risk notes in release artifacts.
• Instrument Observability: Wire pipelines to platform monitoring and defect tracking, with playbooks for alerting, triage, rollback, and hot-fix.
• Document Ingestion and Modeling Choices: Document pattern, orchestration, lineage, and tuning guidance, and keep links to approvals and decisions current through decision tickets and Architecture Decision Records.
• Configure Federated Query Tooling: Configure and implement federated query tooling in alignment with the policies and targets set by the data governance workstream.
• Provide Solution Consulting and Provisioning: Assess customer data, analytics, and operational requirements and recommend fit-for-purpose data architecture and ETL solutions. Guide the provisioning of required data infrastructure, tooling, and pipelines aligned with approved enterprise standards.
• Define and Monitor KPIs: Define and monitor performance and cost KPIs such as timed loads, parallelism, data quality, and storage and egress for chosen patterns, and publish tuning recommendations for domain teams, documented in a Data Quality and Ingestion Dashboard.

What you'll bring:

• Azure Data Factory, Databricks, and Kafka: Experience and expertise with Azure Data Factory, Databricks, and Kafka is required for this role.
• Databricks Unity Catalog: Experience with Unity Catalog for unified governance, security, lineage, and data and AI asset management.
• Data Governance and Access Control Tooling: Familiarity with Collibra and Immuta, or comparable enterprise data catalog and policy-based access control tooling.
• Relevant Disciplinary Expertise: Expertise in the disciplines and technical areas related to the data ingestion, engineering, and quality tasks described above.

Bonus if you have:

• Direct experience working in a federal agency environment

Requirements:

• Applicants must be authorized to work in the United States.
• Bachelor’s degree in a relevant field is required.
• 10+ years of relevant professional experience is required.
• Must be able to obtain and maintain a federal Public Trust (Tier 2 / Moderate Risk) background investigation.
• Must reside and perform all work within the continental United States.
• Production system monitoring and support coverage runs 7:00 a.m. to 8:00 p.m. Eastern Time, with participation in a 24/7 on-call escalation process for urgent issues.
• This position is contingent upon contract award.