Lead Data Engineer

Only local San Francisco Bay Area candidates!

Direct W2 contractors only! No 3rd party agencies!

Hybrid contract: 3 days onsite and 2 days remote in South San Francisco.

No Relocation possible. We don't sponsor any kind of Visa!

As a Lead Data Engineer, you will act as a hands-on technical leader- designing and building data solutions while guiding engineering best practices across the team. This is a player-coach role, where you will actively contribute to development while mentoring others and driving high-quality delivery.

You will be a pivotal member of our team, responsible for:
Key Responsibilities:

• Hands-on Data Engineering & Delivery:
• Design, build, and maintain scalable data pipelines to ingest, transform, and curate structured and unstructured data.
• Write production-quality code in SQL and Python and actively contribute to day-to-day development.
• Troubleshoot, optimize, and improve performance of data workflows and systems.
• Technical Leadership & Data Architecture Ownership:
• Lead by example through hands-on contributions to critical projects.
• Provide technical guidance, code reviews, and mentorship to other data engineers.
• Help drive implementation of best practices in data engineering, testing, and deployment.
• Design and Build Scalable Data Pipelines:
• Architect, develop and oversee development of pipelines to ingest, transform, and curate structured and unstructured data from internal and external sources.
• Ensure high performance, scalability, and reliability of data systems
• Data Profiling, Mapping & Standardization:
• Profile data, identify quality issues, and align disparate datasets. Define data models and standardization frameworks to support scalable, reusable, and AI/ML-ready data products.
• Data Product Engineering & API Development:
• Build and maintain reusable data products and APIs to support analytics and AI use cases.
• Ensure solutions are well-documented, secure, and scalable.
• AI/ML Enablement:
• Work closely with data scientists to prepare and deliver high-quality datasets for ML and AI use cases.
• Support data pipelines for LLM and AI-driven workflows.
• Metadata Management & Data Governance:
• Champion data governance, lineage, and metadata management practices
• Ensure compliance with enterprise data security and privacy standards.
• Monitoring and Event Frameworks:
• Implement monitoring, alerting, and event-driven frameworks for data pipelines.
• Ensure robustness, observability, and reliability of data systems.
• Container and Workflow Orchestration:
• Lead adoption of containerized and orchestrated data workloads (e.g., Docker, Amazon EKS)
• Guide orchestration of complex AI/data workflows using modern tooling.
• Cross-functional Collaboration & Influence
• Partner with business, product, and external stakeholders to align data strategy with organizational goals.
• Translate business needs into scalable technical solutions.
• Continuous Improvement:
• Identify opportunities to improve tooling, processes, and performance.
• Stay hands-on with new technologies and bring practical improvements to the team.

Qualifications Basic Qualifications:

• Bachelor's or Master s degree in Computer Science, Engineering, Data Science, or a related technical field.
• 7+ years of experience in data engineering or similar roles, including experience leading projects or initiatives.
• Proven track record designing and scaling cloud-based data platforms (preferably AWS).
• Strong proficiency in SQL, Python, and advanced data modeling techniques.
• Experience leading architecture decisions and implementing best practices.
• Strong understanding of data quality, integration, transformation, and governance.
• Excellent communication skills with the ability to influence technical and non-technical stakeholders

Preferred Qualifications:

• Experience acting as a technical lead or senior individual contributor on data engineering projects.
• Hands-on experience with AWS data services (Glue, Redshift, S3, Lambda, Athena, etc.).
• Experience supporting AI/ML data pipelines and workflows.
• Familiarity with metadata management and data governance frameworks.
• Experience in healthcare/life sciences or partnering domains.
• Experience working in Agile environments.