Senior Cloud Data Engineer / Data Platform Engineer

12 -18 months contract ; May extend

Remote / Onsite (TBD) - Position is likely to start onsite and could transition to remote depending on level of performance.

General Description:
We are looking for a highly technical engineer to build and support weather data acquisition, processing, and analysis products, including our weather service, using a cloud-based solutions stack. We are open to early or advanced career candidates with strong examples of novel contributions and highly independent work in a fast-paced software delivery environment.

Essential attributes/experience:

Excellent coding skills (Python) with production software deployment experience
Big data experience (terabyte or petabyte level data sources)
Core understanding of cloud computing (e.g., AWS services like IAM, Lambda, S3, RDS)

Example Responsibilities (including but not limited to):

Architect and propose new AWS/Databricks solutions & updates to existing backend systems that ingest, cache, and process terabyte and petabyte level weather data (e.g., from IBM Weather Company API)
Work closely with the product management team and end users to understand customer experience and system requirements, build backlog, and prioritize work
Build infrastructure as code (e.g., Terraform)
Improve system scalability and optimize workflows to reduce cloud costs
Create and update REST APIs and backend processes (e.g., AWS Chalice/Lambda) for current, historical, and forecast weather data
Build/support solutions involving containerization (e.g., Docker), caching (Redis), and databases (e.g., PostgreSQL/PostGIS)
Support Databricks/Spark pipelines for historical weather data backfill into S3 (Parquet/Delta Lake) and data analysis (SQL/PySpark)
Migrate and maintain CI/CD pipelines (e.g., GitHub Actions)
Enhance monitoring and alerting for weather systems (e.g., Datadog)
Enable field testing and customer support operations by debugging and fixing data issues
Work with data scientists to scalably fetch and manipulate large weather data sets for models and analysis