Role: Senior Data Platform Engineer
Location: New York City, NY (3 days Onsite Hybrid)
Prioritization - We need a senior data platform engineer. This engineer must have hands-on experience in Cloud & Data Platform. We prefer experience with Google Cloud Platform and AWS. This engineer builds and operates the platform infrastructure for the data, ML, and analytics teams. The scope includes data governance, security, data pipeline design, CI/CD, and related tools. Platform skills are the first priority: Terraform or a similar IaC tool, IAM and security policies, Spark compute platforms, BigQuery, Trino, job orchestration, and networking. Pipeline engineering and analytics engineering skills are the second priority.
Experience level - This Engineer should have 8+ years of total experience, with 5+ years of hands-on Cloud experience in either AWS or Google Cloud Platform and infrastructure ownership. Please screen out profiles that only design and do not implement the code.
Key Responsibilities:
Must have
Data Processing : Hands-on experience in Spark Jobs, Streaming ingestion, Data Migration and performance tuning (skew, partitioning, autoscaling, cost).
Job Orchestration : Manage and optimize core data platform infrastructure, workflow orchestration (Airflow / Cloud Composer), and automated data quality, observability, and validation frameworks.
Terraform or similar IaC : Infrastructure Design, implementation and CI/CD Process.
On-Call : Own platform SLAs, automated alerting and compute/storage cost optimization.
Data Governance : Dataset access control, policy tags , column-level security, row-level security, cost and quota management.
Hands-on experience in AWS S3, Trino & GCS.
Strong Skills in Python/Java and SQL.
Production ownership: troubleshooting, root-cause analysis and on-call for shared platform services.
Added advantage
Experience with the data migration from AWS to Google Cloud Platform. This includes access between the two clouds, for example Workload Identity Federation and BigQuery Omni.
Data formats: Apache Iceberg, Avro, and Parquet.
Streaming: Kafka, Spark Structured Streaming.
Data governance: PII masking, Cloud DLP, PII Scanner, Apache Ranger.
Kubernetes basics, AWS Lambda, Google Cloud Platform Cloud Run and dbt.
Experience building self-service internal platforms tools for the data users
Google Cloud Platform Professional Cloud Architect or Data Engineer certification.
Collaboration and communication (must have)
Coordinate daily with an offshore engineering team: break down work, write clear tickets and hand-offs, review their code and unblock them across time zones.
Communicate with stakeholders: explain platform decisions, trade-offs and timelines to engineering leads, analytics teams and product partners.
Write clear design docs, runbooks and guides.