Sr. Data Platform & Visualization Engineer

Role: Sr. Data Platform & Visualization Engineer Location: Los Altos, CA (Hybrid) Duration: 12+ months

Important Skills:
Python-focused data platform engineer with experience building reliable ingestion pipelines, SQL-backed metrics systems, backend APIs, and web-based dashboards for ML and experimentation workflows in AWS environments.
• The company collects a lot of data on a monthly trip and potentially from simulations.
• Currently, they are not handling much of it.
• The main target is that their monthly trip collects a large amount of vehicle data.
• Previously, it used to be mostly just vehicle numbers.
• They are starting to add a lot of camera and LIDAR data, which means that
• the scale of the data is getting larger.
• Looking for someone junior to help with the existing ingestor system that ingests the
• data and uploads it to S3.
• Bandwidth is very limited (100 Mbps) because they go to a remote location, and
• they are trying to upload 500 GB of data per night.
• They want some sort of orchestration logic for sequencing and prioritizing certain
• steps.
• Once the data is up there, they are looking for someone with front-end experience to
• navigate, filter, and create metadata.
• This will require some back-end experience as well to handle the data.
• Current Process:
• It's not very smooth, but they make it work.
• If they cannot finish uploading the data on the spot, they have to wait hours and
• then come back to the office where there is fast data and upload it.
• Hoping to streamline and optimize the process.

Clarification:
• There is a queuing system where they have a bunch of stuff stored locally and
• need to peter it out to the cloud storage.
• That system needs to be refined and maintained.