Senior Data Engineer (Spark, Kafka & Google Cloud Platform)
Pacific Consultancy Services
Dice Job Match Score™
👾 Reticulating splines...
Job Details
Skills
- spark
- pyspark
- apache airflow
- etl process
- big data
- GCP
Summary
Senior Data Engineer (Spark, Kafka & Google Cloud Platform)
Sunnyvale, CA (Onsite – 5 Days)
Contract
UST Global // Walmart
22857-1
Mandatory Areas
Must Have Skills – Data Engineer
Skill 1 – big data, spark, pyspark, apache airflow, etl process
Skill 2 – 4+ years in Google Cloud Platform
Job Summary
We are looking for a highly experienced Senior Data Engineer with strong expertise in Apache Spark, PySpark, Kafka, and Google Cloud Platform (Google Cloud Platform) to build scalable real-time and batch data processing solutions. The ideal candidate will have extensive experience designing and optimizing enterprise data pipelines for analytics and machine learning workloads.
Required Skills
- 10+ years of overall IT experience
- 4+ years of recent hands-on experience with Google Cloud Platform (Google Cloud Platform)
- Strong programming skills in Python and SQL
- Experience with Scala or Java
- Expertise in Apache Spark (Spark SQL, DataFrames, Spark Streaming)
- Hands-on experience with PySpark
- Strong knowledge of Apache Kafka or Google Pub/Sub
- Experience building ETL/ELT pipelines
- Experience with BigQuery, Data Lakes, and Data Warehouses
- Knowledge of Snowflake or Amazon Redshift
- Experience with Apache Airflow
- Good understanding of data quality, monitoring, and performance tuning
- Experience with Databricks, Docker, and Kubernetes is a plus
Responsibilities
- Design, develop, and maintain scalable batch and real-time ETL/ELT pipelines.
- Build streaming data pipelines using Spark and Kafka.
- Develop optimized data solutions for analytics and machine learning.
- Optimize Spark jobs, SQL queries, and overall data processing performance.
- Implement data quality checks, monitoring, and alerting mechanisms.
- Work with cloud-native data services on Google Cloud Platform.
- Collaborate with cross-functional teams to deliver scalable data engineering solutions.
- Participate in design reviews, code reviews, and production deployments.
- Troubleshoot and resolve production issues while ensuring high availability.
Preferred Qualifications
- Experience working in large enterprise environments.
- Strong understanding of Agile/Scrum methodologies.
- Excellent communication and problem-solving skills.
- Experience mentoring junior engineers is preferred.
Nice to Have
- Databricks
- Docker
- Kubernetes
- BigQuery
- Snowflake
- Redshift
- Google Pub/Sub
- Dice Id: 91142718
- Position Id: 9036872
- Posted 5 hours ago
Company Info
Pacific Consultancy Services, founded with an ambitious vision in 2013, is a prominent IT Consulting and Service Delivery firm. The company is built upon the pillars of exceptional customer-centric solutions, streamlined processes, and impeccable technical expertise.
With nearly two decades of experience, Pacific consultancy services have been at the forefront of delivering intelligent solutions to clients worldwide, including the United States. Its specialized offerings encompass Artificial Intelligence, Machine Learning, Blockchain, Cloud services, IoT, DevOps, IT Staff Augmentation, and Cognitive Analytics, all contributing to achievable and profitable business models. Ensuring quality across all service domains is a priority for Pacific consultancy services. Its services span from IT Staff Augmentation to Digital Transformation, IT Consulting, and Emerging Technologies.
The work ethos of Pacific consultancy services revolves around its core “Model of Delivery,” aimed at providing clients with the best possible solutions.
Never miss an opportunity! Create an alert based on the job you applied for.Similar Jobs
It looks like there aren't any Similar Jobs for this job yet.
Search all similar jobs