Job Description
Data Engineer - Google Cloud Platform / PySpark
Job Summary
We are looking for an experienced Data Engineer with strong hands-on expertise in Google Cloud Platform (Google Cloud Platform), Apache Kafka, Apache Airflow, PySpark, and Hive. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines, processing large volumes of data, and supporting batch and real-time data processing requirements.
Key Responsibilities
Design, develop, and maintain scalable data pipelines using Google Cloud Platform services and Apache Spark.
Build and optimize data processing workflows using Python, PySpark, and Spark.
Develop and maintain real-time data streaming pipelines using Apache Kafka.
Create, schedule, and monitor data workflows using Apache Airflow and DAGs.
Work with Hive for querying, transforming, and managing large datasets.
Implement efficient ETL/ELT processes for data ingestion, transformation, and integration.
Optimize data pipelines for performance, scalability, reliability, and cost efficiency.
Troubleshoot data processing issues and ensure data quality and consistency.
Collaborate with cross-functional teams to understand data requirements and deliver reliable data solutions.
Required Skills
Strong hands-on experience with Google Cloud Platform (Google Cloud Platform).
Experience with Apache Kafka and real-time data streaming.
Proficiency in Apache Airflow for workflow orchestration and scheduling.
Strong programming skills in Python and experience with PySpark or Apache Spark.
Experience working with Apache Hive and large-scale data processing.
Strong understanding of distributed computing, data engineering concepts, and ETL/ELT pipelines.
Nice-to-Have Skills
Experience with Scala programming.
Familiarity with additional Google Cloud Platform data services such as BigQuery, Google Cloud Storage, and Dataproc.
Experience with data pipeline performance tuning and optimization.
Preferred Candidate Profile
The ideal candidate will have practical experience building and supporting data pipelines in a cloud environment, processing large datasets using Spark, implementing streaming solutions with Kafka, and orchestrating workflows through Airflow. Strong Google Cloud Platform, PySpark/Spark, Kafka, Airflow, and Hive skills are essential. Scala is an added advantage.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: wesca004
- Position Id: JOB-7421
- Posted 14 hours ago
