Senior PySpark Developer

Senior PySpark Developer

Location: Iselin, NJ

Contract: 6-12 Months

We are seeking a highly skilled Senior PySpark Developer to join our data engineering team. The ideal candidate will have extensive experience in building scalable data pipelines using Apache Spark with Python (PySpark), and a strong background in big data technologies, data modeling, and performance optimization.

Key Responsibilities:

• Design, develop, and maintain large-scale data processing systems using PySpark.
• Build and optimize data pipelines and ETL processes for structured and unstructured data.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements.
• Implement data quality checks and ensure data integrity across systems.
• Optimize Spark jobs for performance and scalability in distributed environments.
• Work with cloud platforms (e.g., AWS, Azure, Google Cloud Platform) and data lake architectures.
• Participate in code reviews, testing, and deployment processes.
• Mentor junior developers and contribute to best practices in data engineering.

Required Skills & Qualifications:

• Bachelor s or Master s degree in Computer Science, Engineering, or a related field.
• 10+ years of experience in data engineering or software development.
• 3+ years of hands-on experience with PySpark and the Apache Spark ecosystem.
• Strong proficiency in Python and SQL.
• Experience with big data tools and frameworks (e.g., Hadoop, Hive, Kafka).
• Familiarity with cloud services (AWS EMR, Azure Databricks, Google Cloud Platform DataProc).
• Solid understanding of data warehousing concepts and data modeling.
• Experience with CI/CD pipelines and version control (e.g., Git).
• Excellent problem-solving and communication skills.