Job Description
Data Engineer
Location: Sunnyvale, CA / Bentonville, AR
Work Model: Onsite - 5 days/week
Duration: Contract
Job Description
We are looking for an experienced Data Engineer to design, develop, and maintain scalable data pipelines and data processing solutions supporting large-scale enterprise platforms. The ideal candidate will have strong hands-on experience with Python, PySpark, SQL, cloud data platforms, distributed data processing, and data pipeline development.
The Data Engineer will work closely with data scientists, software engineers, product teams, architects, and other stakeholders to build reliable, scalable, and high-performance data solutions.
Required Skills
5+ years of experience in Data Engineering or a closely related field
Strong programming experience with Python
Strong hands-on experience with Apache Spark / PySpark
Strong expertise in SQL, including complex queries, optimization, and data transformations
Experience designing and developing ETL/ELT data pipelines
Experience working with Databricks or similar distributed data processing platforms
Strong experience with cloud data platforms, preferably Azure or Google Cloud Platform
Experience with cloud storage and data services such as Azure Data Lake, ADLS, GCS, BigQuery, or equivalent
Experience with Apache Kafka or other event-streaming technologies
Experience with Airflow / Cloud Composer or similar workflow orchestration tools
Strong understanding of data modeling, data warehousing, and data architecture
Experience working with both relational and NoSQL databases
Experience with Git and CI/CD
Strong understanding of data quality, monitoring, testing, and pipeline reliability
Responsibilities
Design, develop, and maintain scalable batch and real-time data pipelines
Build ETL/ELT workflows using Python, PySpark, SQL, and cloud technologies
Develop distributed data processing solutions using Apache Spark/Databricks
Build data pipelines integrating data from multiple sources and systems
Develop streaming and event-driven data pipelines using Kafka
Implement workflow orchestration using Airflow or similar technologies
Design and optimize data models, queries, and data processing jobs
Work with cloud data platforms such as Azure and Google Cloud Platform
Implement data quality checks, validation, monitoring, and error handling
Troubleshoot pipeline failures and performance issues
Optimize Spark jobs, SQL queries, and data processing workflows for performance and scalability
Collaborate with data scientists, software engineers, product managers, architects, and DevOps teams
Participate in technical design, code reviews, testing, deployment, and production support
Follow engineering best practices for security, scalability, reliability, and maintainability
Preferred Skills
Experience with Databricks
Strong experience with PySpark and Python
Experience with Scala is a plus
Experience with Google Cloud Platform, BigQuery, GCS, Dataproc, and Cloud Composer
Experience with Azure, ADLS, Azure Data Factory, and Azure Databricks
Experience with Kafka
Experience with Airflow
Experience with Snowflake
Experience with PostgreSQL or other relational databases
Experience with NoSQL databases
Experience with Terraform, Docker, or Kubernetes
Experience developing real-time/streaming data pipelines
Experience working with large-scale datasets and high-volume data processing
Experience in retail, e-commerce, marketplace, or other large-scale enterprise environments
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
- Dice Id: wesca004
- Position Id: JOB-7398
- Posted 1 hour ago
Never miss an opportunity! Create an alert based on the job you applied for.
