Pyspark Engineer

Phoenix, AZ, US • Posted 5 hours ago • Updated 5 hours ago

Full Time

On-site

Fitment

Dice Job Match Score™

✨ Finding the perfect fit...

Job Details

Skills

Data Engineering
Data Processing
SQL
Python
Data Integrity
Analytical Skill
Machine Learning (ML)
Extract
Transform
Load
Apache Hadoop
Databricks
Amazon Web Services
Unstructured Data
Data Quality
Collaboration
Performance Tuning
Cost Reduction
Scalability
Data Security
Apache Spark
Distributed Computing
Workflow
Apache Airflow
cron
Orchestration
PySpark
Big Data
Cloud Computing

Summary

We are seeking a skilled PySpark Engineer to join our data engineering team. The ideal candidate will have expertise in Apache Spark (PySpark) for large-scale data processing, strong SQL and Python skills, and experience working with big data platforms such as Hadoop, Databricks, or AWS Glue. This role involves building scalable ETL pipelines, optimizing data workflows, and ensuring data integrity for analytical and machine learning applications.

Key Responsibilities:

Develop and optimize ETL pipelines using PySpark for data ingestion, transformation, and processing.
Design, implement, and maintain big data solutions on platforms such as Hadoop, Databricks, or AWS Glue.
Work with structured and unstructured data from multiple sources, ensuring data quality and consistency.
Collaborate with data scientists, analysts, and business teams to understand data requirements and deliver efficient solutions.
Optimize Spark jobs for performance tuning, cost reduction, and scalability.
Implement data security and governance best practices.
Troubleshoot issues related to data pipelines, Spark jobs, and distributed computing environments.
Automate workflows using Apache Airflow, Cron Jobs, or other orchestration tools.
Write clean, maintainable, and well-documented PySpark code following best practices.
Stay up to date with the latest developments in big data, cloud computing, and distributed systems.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.

Dice Id: 80183517
Position Id: 873a9d2ee77bccc7a4f9d1bec745f9ad
Posted 5 hours ago

Create job alert

Never miss an opportunity! Create an alert based on the job you applied for.

Phoenix, Arizona

•

Today

Job Description Senior Java Data Engineer Enterprise Data Frameworks Role Summary The Enterprise Data Frameworks team is seeking a Senior Java focused software engineer who builds and maintains large scale data processing systems using Java, Apache Spark, and Kafka. This role is intended for experienced backend engineers with strong core Java fundamentals who apply traditional software engineering practices to data intensive platforms and distributed systems. The ideal candidate has hands on

Full-time

USD 105,000.00 - 158,000.00 per year

Data Pipeline Engineer (Databricks + AWS Glue)

Remote

•

4d ago

Location: United States (Remote / Hybrid)Job Type: Full-TimeExperience: 510 Years About the RoleWe are seeking a highly skilled Data Pipeline Engineer with expertise in Databricks, AWS Glue, PySpark, Python, SQL, MongoDB, and AWS to build scalable, cloud-native data platforms. You will be responsible for designing, developing, and optimizing high-performance ETL pipelines that process large volumes of structured and semi-structured data for analytics, reporting, and enterprise applications. This

Easy Apply

Full-time

Depends on Experience

Databricks Administrator

Remote

•

11d ago

Job Title: Databricks AdministratorJob SummaryWe are looking for a Databricks Admin with strong hands-on experience in Databricks, AWS, and modern data platforms. The ideal candidate should have experience designing and supporting scalable data pipelines, administering the Databricks platform, optimizing performance and cloud costs, implementing governance, and troubleshooting production issues. Key ResponsibilitiesDesign, build, and maintain scalable data pipelines using AWS and Databricks.Deve

Easy Apply

Full-time

$110,000 - $120,000

Databricks Administrator

Remote

•

11d ago

Easy Apply

Full-time

$110,000 - $120,000

Search all similar jobs

More jobs at Texas State Library and Archives Commision in Phoenix, AZ