We are seeking a Senior Databricks Solution Architect to lead the architecture and technical execution of a Palantir Foundry to Databricks migration. The role involves assessing existing Palantir applications, mapping capabilities to Databricks services, defining the target-state Lakehouse architecture, and leading the migration of data engineering, AI/GenAI, analytics, chatbots, and business workflows into scalable, secure, and high-performance solutions.
Key Responsibilities
· Define the target-state Databricks architecture, including scalable Lakehouse design across Bronze, Silver, and Gold layers.
· Assess existing Palantir Foundry pipelines, datasets, applications, workflows, and AI capabilities; develop feature-by-feature mappings to Databricks and identify areas requiring redesign.
· Lead the migration of data ingestion, transformation, data quality, validation, large-scale data comparison, and reconciliation capabilities.
· Design and optimize data pipelines using Databricks Lakeflow, Spark Declarative Pipelines, Apache Spark, and Delta Lake, supporting batch and near-real-time processing.
· Architect AI/GenAI solutions for structured and unstructured document extraction, LLMs, RAG, embeddings, vector search, model serving, and migration of Palantir AIP capabilities.
· Design target-state solutions for dashboards, drill-down reporting, analytical applications, chatbots, conversational analytics, and business workflows.
· Establish enterprise data governance, security, access controls, lineage, and auditing using Unity Catalog and appropriate RBAC policies.
· Optimize Databricks workloads, data pipelines, and compute resources for scalability, performance, reliability, and cost efficiency.
· Lead data engineering and AI/ML teams, conduct architecture reviews, prepare HLD/LLD documents and migration roadmaps, and collaborate with business stakeholders, Palantir SMEs, cloud architects, and application teams.
·
Requirements
· 15+ years of IT experience.
· Extensive experience in Databricks solution architecture, Lakehouse design, Delta Lake, Apache Spark, and enterprise data engineering.
· Strong expertise in Bronze/Silver/Gold architecture, data ingestion, transformation, data quality, CDC, and batch/near-real-time processing.
· Proven experience with Palantir Foundry and/or Palantir AIP, preferably involving platform migration, capability mapping, and application or workflow redesign.
· Strong knowledge of AI/GenAI solutions, including LLMs, RAG, document intelligence, information extraction, embeddings, vector search, and model serving.
· Hands-on experience with Unity Catalog, enterprise data governance, security, access controls, and cloud platforms such as AWS, Azure, or Google Cloud Platform.
· Experience architecting analytics, dashboards, chatbots, application integrations, and business workflows in enterprise environments.
· Demonstrated technical leadership, architecture documentation, performance optimization, stakeholder management, and ability to lead complex migration initiatives.
· Bachelors degree in Computer Science, Engineering or related.
