Location: Dallas, TX/Seattle, WA/Atlanta, GA/NY, NJ (Hybrid Work)
We are seeking a highly experienced Lead Data Engineer to architect, develop, and optimize scalable data solutions using Palantir Foundry. This senior role is ideal for a data engineering expert who thrives in complex, ontology-aware environments and can lead the design of robust data pipelines and applications that drive actionable insights. You will play a pivotal role in shaping our data strategy, collaborating with stakeholders across the organization, and ensuring the integrity, performance, and cost-efficiency of our data ecosystem. While familiarity with Generative AI (GenAI) is a plus, your core expertise should lie in data architecture, pipeline development, and Foundry platform mastery.
Key Responsibilities
Data Architecture & Pipeline Engineering
• Design, build, and maintain end-to-end ETL/ELT pipelines using PySpark, Spark SQL, and Python.
• Architect scalable data frameworks that support high-performance analytics and operational workflows.
• Conduct data cleansing, transformation, and validation to ensure data quality and consistency.
Palantir Foundry Platform Expertise
• Configure and manage data connections, Pipeline Builder, and Contour for data exploration and visualization.
• Develop and maintain ontology objects, ensuring semantic consistency and reusability across applications.
• Manage code repositories and version control within Foundry, promoting modular and maintainable engineering practices.
Ontology-Aware Application Development
• Collaborate with data owners and domain experts to ingest, transform, and model data into ontology-driven applications such as Workshop and Quiver.
• Maintain clarity on the end-to-end architecture and data flow, ensuring seamless integration from ingestion to ontology.
Cross-Functional Collaboration & Leadership
• Partner with data scientists, analysts, and business stakeholders to translate requirements into scalable data solutions.
• Engage with senior leadership to maintain a feedback loop for Foundry program improvements.
• Provide mentorship and technical guidance to junior engineers and analysts.
Operational Excellence & Cost Management
• Establish and maintain cost monitoring and reporting for Foundry-related operations.
• Lead efforts in troubleshooting, performance tuning, and platform stability.
Continuous Improvement & Innovation
• Stay current with Foundry platform updates, emerging features, and best practices.
• Identify opportunities for automation, optimization, and innovation in data workflows.
• Explore and evaluate GenAI capabilities for potential integration into data processes.
Never miss an opportunity! Create an alert based on the job you applied for.
