Python\Pyspark Engineer

Python\Pyspark Engineer

Location Jersey City, NJ

Start ASAP

Job Description:

Technical Skills

• Strong proficiency in Python, packaging, dependency management, and virtual environments.
• Hands-on experience with PySpark, including Spark performance tuning (partitioning, caching, broadcast joins, memory optimization).
• Expertise in data ingestion (batch/stream), schema management, and robust error-handling/retry logic.
• Solid unit and integration testing practices, including data quality validations.
• Experience with CI/CD pipelines (Azure DevOps/Jenkins), Git branching strategies, and artifact versioning.
• Working experience with Cloudera/Hadoop (HDFS, Spark, Hive/Impala) and Databricks (Delta Lake, clusters, jobs, notebooks).
• Knowledge of observability techniques: structured logging, metrics, tracing, and debugging in distributed systems.
• Secure coding practices including secrets management, PII protection, and compliance-aware development.

Functional Skills

• Strong documentation discipline for frameworks, reusable components, and best-practice patterns.
• Effective collaboration with Cloud Architects and Data Ops to ensure stable and supportable pipelines.
• Clear communication of technical ideas and solution approaches.
• Comfort working in Agile environments with iterative development and frequent releases.

Roles & Responsibilities

• Design and develop scalable Python/PySpark ingestion and transformation pipelines.
• Implement schema evolution logic, validation frameworks, and resilient error-handling mechanisms.
• Optimize Spark jobs for performance, cost efficiency, and production readiness.
• Integrate all jobs into automated CI/CD pipelines, ensuring versioning and release governance.
• Work closely with Ops teams to ensure proper monitoring, logging, and operational supportability.
• Participate in Agile ceremonies, sprint planning, code reviews, and demo sessions