Python Engineer, PySpark/Spark Performance Tuning, Azure DevOps/Jenkins, Hive/Impala; NJ

Start: ASAP
Glider: Candidate Identification Only
3 days Onsite Required
Jersey City, NJ - NJ2-101-06-01.
MUST INCLUDE CURRENT LOCATION AND INTENDED ON LOCATION RESUME

The Python Engineer designs and implements data ingestion and transformation jobs using Python and PySpark. They build reusable frameworks for data quality checks and integrate pipelines with CI/CD processes. The role requires performance tuning, secure coding practices, and strong debugging skills. Close collaboration with architects and DataOps teams ensures robust and compliant solutions.

Required Qualifications
The Python Engineer designs and implements data ingestion and transformation jobs using Python and PySpark.
They build reusable frameworks for data quality checks and integrate pipelines with CI/CD processes.
The role requires performance tuning, secure coding practices, and strong debugging skills. Close collaboration with architects and Data Ops teams ensures robust and compliant solutions.

Desired Qualifications
Strong Python (packaging, virtual environments), PySpark/Spark performance tuning.
Experience with data ingestion (batch/stream), schema management, and error handling/retry logic.
Test discipline: unit/integration tests, data quality assertions, reproducible pipelines.
CI/CD (Azure DevOps/Jenkins), Git workflows, artifact versioning, release readiness.
Experience on Cloudera/Hadoop (HDFS, Spark, Hive/Impala) and Databricks (clusters, jobs, notebooks, Delta).
Observability: structured logging, metrics, tracing; debuggability in distributed contexts.
Secure coding and handling of secrets, PII protection, and compliance considerations.
Strong communication and collaborative work style; documentation of frameworks and usage patterns.
Performance optimization: partitioning, caching, broadcast joins, memory tuning.
Comfort with Agile ceremonies and iterative delivery.
Min 5 yrs of experience

Enable Skills-Based Hiring No

Qualification Assessment
Must Have
Technical
Python
and 0 years
Technical by default
Microsoft Azure
and 0 years
Spark
and 0 years