Lead AI Engineer, Agentic Data Lineage

Lead AI Engineer, Agentic Data Lineage

Location: Montreal, Canada

Work Type: Hybrid min 3 days work from office per week

Interview rounds: L1 and L2 

JD:

Lead AI Engineer, Agentic Data Lineage

What You Will Do:

• Define and drive the technical strategy for a Firmwide, column-level data lineage platform built on AI reasoning agents.
• Architect parsers that build Abstract Syntax Trees (ASTs) from SQL across multiple dialects, stored procedures, Python, Java, Scala, and Spark jobs, then resolve column-level transformations from those trees.
• Design ingestion of no-code signals from ETL and transformation tools, workflow orchestrators, data catalogs, and BI platforms, normalizing them into a common lineage model.
• Build a multi-agent system in which specialized deep agents parse, reason, disambiguate, and reconcile conflicting signals, then assemble a coherent end-to-end lineage graph.
• Establish how agents handle ambiguity (dynamic SQL, runtime configuration, hand-written transformations) by reasoning over context rather than failing silently or guessing.
• Model and persist lineage as a queryable knowledge graph supporting impact analysis, root-cause tracing, and regulatory evidence generation.
• Build confidence scoring and validation so every lineage edge carries a provenance signal and a measure of certainty, with low-confidence inferences surfaced for human review.
• Partner with the Firmwide Data Office, Risk, Compliance, and the divisional technology teams to align the platform with the Firm''s data governance obligations, including BCBS 239, CCAR, GDPR, and SOX.
• Lead, mentor, and grow a team of AI and data engineers, setting engineering standards, reviewing designs, and owning delivery.
• Operate within Morgan Stanley''s controls for AI safety, data confidentiality, and model governance.

   
Required Qualifications:

• 8+ years of software and data engineering experience, including a track record of leading complex platform builds end to end.
• Strong background in program analysis or compiler techniques: parsing, ASTs, intermediate representations, and static analysis. Hands-on experience with tools such as ANTLR, tree-sitter, sqlglot, or language-native AST libraries.
• Practical experience building LLM-powered agents or multi-agent systems, including reasoning loops, tool use, retrieval, and orchestration frameworks (LangGraph, LlamaIndex, AutoGen, or equivalent in-house systems).
• Deep data engineering expertise: SQL fluency across dialects, plus experience with Spark, distributed pipelines, and modern transformation tooling such as dbt.
• Experience modeling and querying graph data, with a graph database such as Neo4j, Amazon Neptune, or Janus Graph.
• Strong software engineering fundamentals: testing, CI/CD, and production-grade systems at scale.
• Demonstrated technical leadership, including mentoring engineers and influencing architecture across teams.

Preferred Qualifications:

• Experience in financial services, capital markets, or another heavily regulated data environment.
• Familiarity with data lineage and metadata standards and tools such as OpenLineage, Marquez, DataHub, Apache Atlas, Egeria, Collibra, or Alation.
• Working knowledge of BCBS 239 data aggregation and risk reporting principles and how they translate into engineering requirements.
• Experience harvesting metadata from enterprise ETL tools (Informatica, Talend) and BI platforms (Tableau, Power BI).
• Background in knowledge representation, semantic reasoning, or evaluation of LLM output quality and hallucination control.

   
Tech Stack:
Python, SQL, Spark, and Scala or Java for parsing targets. LLM and agent frameworks for reasoning. Graph databases for lineage storage. Workflow orchestration for pipeline execution. Cloud and on-premise deployment with containerized services. OpenLineage-aligned metadata models.