Google Cloud Platform Data Engineer

Role Overview

• We are seeking a Senior / Lead Data Engineer with strong hands-on technical depth, data architecture and design experience, and proven technical leadership skills.
• This role will play a key part in modernizing the enterprise data platform, leading complex data engineering initiatives, and guiding design decisions as workloads transition from on-prem to Google Cloud Platform (Google Cloud Platform).
• The ideal candidate is comfortable operating as a technical lead, owning end-to-end data solutions, mentoring team members, and partnering closely with product owners and stakeholders.

Key Responsibilities

• Design and develop ETL/ELT workflows and data pipelines for batch and real-time processing.
• Build and maintain data pipelines for reporting and downstream applications using opensource frameworks and cloud technologies.
• Implement operational and analytical data stores leveraging Delta Lake and modern database concepts.
• Optimize data structures for performance and scalability across large datasets.
• Collaborate with architects and engineering teams to ensure alignment with target state architecture.
• Apply best practices for data governance, lineage tracking, and metadata management, including integration with Google Data plex for centralized governance and data quality enforcement.
• Develop, schedule, and orchestrate complex workflows using Apache Airflow, with strong proficiency in designing and managing Airflow DAGs.
• Troubleshoot and resolve issues in data pipelines and ensure high availability and reliability.

Required Technical Skills

• Strong Understanding of Data: Data structures, modeling, and lifecycle management.
• ETL/ELT Expertise: Hands-on experience designing and managing data pipelines.
• PySpark: Advanced skills in distributed data processing and transformation.
• NetApp Iceberg: Experience implementing open table formats for analytics.
• Hadoop Ecosystem: Knowledge of HDFS, Hive, and related components.
• Cloud Platforms: Google Cloud Platform (Big Query, Dataflow), Delta Lake, and data plex for governance and metadata management.
• Programming & Orchestration: Python, Spark, SQL.
• Workflow Orchestration: Strong experience with Apache Airflow, including authoring and maintaining DAGs for complex workflows.
• Database & Reporting Concepts: Strong understanding of relational and distributed systems.

The project focuses on modernizing and simplifying the enterprise data technology stack by establishing centralized ODS and ADS platforms within HL.

• Data for multiple L2 products currently spans 30+ applications
• Used for both operational and analytical purposes
• Results in data redundancy, complexity, and high maintenance costs
• This role will be central to reducing complexity and driving cloud-first architecture