Must Have Skills:
· Databricks - Spark (PySpark or Spark SQL, Delta Lake (medallion architecture), Unity Catalog (Multi-tenant)
· Python & SQL
· Experience owning multi-tenant data architecture
· Experience in Higher Education technology (CRM/SIS/Degree Audit platforms, etc.)
· Working knowledge of FERPA, GLBA, SOC 2
· Data Security and Governance
· GitHub, working with a CI/CD
· Architecture Reviews, decisions
· Evaluating emerging tools, models and frameworks
Nice to Have Skills
· Exposure to agentic AI systems or LLM adjacent data pipelines.
· Experience with independently standing up net-new Unity Catalog structures for new tenants from scratch.
Job Responsibilities
· Design, build and maintain Databricks pipelines across bronze/silver/gold layers for multiple higher education clients unifying data across the institution, each within its own Unity Catalog.
· Create and implement strategies for moving client data from their CRM/SIS platforms via API calls or into drop zones (SFTP) for ingestion into Databricks. Examples include Slate SFTP feeds, Banner derived warehouses, EAB Navigate 360 and Stellic, all core higher education platforms, with attention to schema drift and data quality.
· Own the thin, tenant scoped physical ingest versus a Lakehouse Federation decision for new client onboarding.
· Implement Data governance controls including Unity Catalog lineage and audit logging. Enforce PII de-identification gates and FERPA compliance across patterns.
· Be the resident expert on changing compliance requirements and plan as needed while keeping all the stakeholders informed.
· Partner with the AI Solutions team to translate agent data requirement into concrete schemas and feature tables.
· Support the onboarding of new institutional clients, each with their distinct CRM/SIS stacks and data dictionaries.
· Contribute and suggest reusability for components, especially with connectors so that patterns build for one client can be reused across the portfolio.
· Work with engineering as the primary engineering/infrastructure partner on platform level Databricks and AWS decisions
Required Experience:
· At least 3 years of building production data pipelines, with hands-on Databricks experience (Spark, Delta Lake, Unity Catalog, etc.).
· Strong SQL and Python skills with familiarity working across batch ETL/ELT patterns.
· Experience with multi-tenant data architecture and a deep understanding of catalog isolation reasoning.
· Working knowledge of FERPA, GLBA, SOC 2 or equivalent regulated-data experience (higher education, healthcare, financial services).
· Experience with AWS (our deployment model is single-tenant private AWS per client).
· Comfortable working with a geographically dispersed team across the world.
Educational Qualifications:
· Bachelor's/Master's in Computer Science, AI, Engineering or related field.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: 91156852
- Position Id: 9143244
- Posted 1 day ago