General Summary:
The Staff Data Engineer / Full?Stack Data Developer is a senior, hands?on individual contributor responsible for designing, building, optimizing, and operating data pipelines, curated data products, and Databricks?native data applications on a modern cloud Lakehouse platform. This role is critical to enabling enterprise analytics, BI, AI/ML, and data?driven applications, with deep expertise in Databricks, Python, Spark, and Databricks application development.
This position requires strong end?to?end ownership of data engineering and data app solutions, production?grade engineering rigor, and the ability to collaborate across platform, analytics, and application teams.
This role requires full-time onsite work in San Diego, CA (5 days per week).
Minimum Qualifications:
• 5+ years of IT-related work experience with a Bachelor''s degree in Computer Engineering, Computer Science, Information Systems or a related field.
OR
7+ years of IT-related work experience without a Bachelor’s degree.
• 3+ years of work experience with programming (e.g., Java, Python).
• 3+ years of work experience with SQL or NoSQL Databases.
• 3+ years of work experience with Data Structures and algorithms.
Key Responsibilities
Data Engineering & Development
Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, and Python to support enterprise analytics, AI, and application use cases.
Build and manage curated data layers following Lakehouse Medallion architecture best practices (Bronze / Silver / Gold).
Develop reusable, modular data transformation frameworks to accelerate delivery across domains.
Databricks Application Development
Design and develop Databricks?native data applications, including notebook?based apps, Databricks dashboards, and interactive data experiences for analytics and business users.
Build data APIs, parameterized pipelines, and app?integrated data services leveraging Databricks and Lakehouse capabilities.
Partner with analytics, AI, and application teams to embed data and insights directly into workflows and applications.
Ensure Databricks apps meet performance, security, governance, and usability standards.
Performance, Scalability & Reliability
Optimize Apache Spark jobs and Databricks workloads for performance, cost efficiency, scalability, and reliability.
Proactively address challenges related to data volume, schema evolution, and compute optimization.
Implement robust data quality checks, validations, and anomaly detection within pipelines and apps.
Production Support & Operations
Own and support production data pipelines and Databricks applications, including monitoring, troubleshooting, and root?cause analysis.
Ensure high availability, data correctness, and SLA adherence for business?critical datasets and apps.
Contribute to observability, alerting, and operational automation.
Full?Stack Data Enablement
Collaborate with BI, analytics, AI/ML, platform, and application teams to deliver end?to?end data solutions.
Enable data consumption across dashboards, reports, Databricks apps, AI models, APIs, and downstream applications.
Translate business and analytical requirements into well?designed data pipelines and data applications.
Engineering Excellence & Technical Influence
Act as a technical leader and mentor, defining best practices for data engineering and Databricks app development.
Participate in architecture reviews, design discussions, and technical roadmaps.
Continuously evaluate and adopt modern Databricks features, GenAI capabilities, and automation patterns to improve developer productivity.
Required Skills & Experience
5+ years of hands?on data engineering experience, owning production?grade pipelines and data solutions.
Strong proficiency in Python and Apache Spark (PySpark).
Proven hands?on experience working with Databricks in production, including Databricks application development.
Strong SQL and data transformation skills.
Experience building and supporting Databricks notebooks, dashboards, and data?driven applications.
Experience operating and supporting data pipelines and data apps in production environments.
Solid understanding of data quality, reliability, security, and governance.
Preferred / Nice?to?Have Qualifications
Experience with AWS cloud services (e.g., S3, IAM, EC2, Glue, or equivalent).
Exposure to Unity Catalog, access controls, metadata management, and governed data sharing.
Experience with streaming data pipelines (e.g., Structured Streaming, Kafka).
Familiarity with CI/CD, Git?based workflows, and Data/Analytics DevOps.
Experience enabling BI, AI/ML, or application?embedded analytics using Databricks.
What Defines Success at the Staff Level
Owns complex data pipelines and Databricks applications end?to?end with minimal oversight.
Drives improvements in performance, reliability, cost efficiency, and usability across data and app layers.
Influences architecture, standards, and best practices beyond immediate assignments.
Serves as a trusted technical partner to analytics, AI, platform, and application teams.
Determining compensation for this role (and others) at Vaco/Highspring depends upon a wide array of factors including but not limited to the individual’s skill sets, experience and training, licensure and certifications, office location and other geographic considerations, as well as other business and organizational needs. With that said, as required by local law in geographies that require salary range disclosure, Vaco/Highspring notes the salary range for the role is noted in this job posting. The individual may also be eligible for discretionary bonuses, and can participate in medical, dental, and vision benefits as well as the company’s 401(k) retirement plan. Additional disclaimer: Unless otherwise noted in the job description, the position Vaco/Highspring is filing for is occupied. Please note, however, that Vaco/Highspring is regularly asked to provide talent to other organizations. By submitting to this position, you are agreeing to be included in our talent pool for future hiring for similarly qualified positions. Submissions to this position are subject to the use of AI to perform preliminary candidate screenings, focused on ensuring minimum job requirements noted in the position are satisfied. Further assessment of candidates beyond this initial phase within Vaco/Highspring will be otherwise assessed by recruiters and hiring managers. Vaco/Highspring does not have knowledge of the tools used by its clients in making final hiring decisions and cannot opine on their use of AI products.
Never miss an opportunity! Create an alert based on the job you applied for.