PySpark Developer (Cloudera, Iceberg, Airflow & AWS)

Job Title: Senior PySpark Developer
Location: Remote – US
Duration: 12 Months
Positions: 1
Experience: 10–12+ Years
Level: Senior / Lead Developer

Job Description:

Seeking a Senior PySpark Developer with strong hands-on experience in Cloudera CDP, Apache Iceberg, Airflow, and AWS to build and optimize large-scale data engineering solutions.

Mandatory Skills:

• 10+ years of overall IT experience with 6+ years in Data Engineering.

• Strong hands-on experience with PySpark and Apache Spark.

• Cloudera Data Platform (CDP).

• Apache Iceberg and Lakehouse architecture.

• Apache Airflow for workflow orchestration.

• Strong SQL, ETL/ELT, and data modeling experience.

• Large-scale data processing and pipeline development.

• Data partitioning, performance tuning, and query optimization.

• Git, CI/CD, and DevOps practices.

Good to Have:

• AWS services: S3, Glue, Lambda, EMR, ECS/EKS, RDS, IAM, and CloudWatch.

• Kafka and event-driven architecture.

• Data quality, metadata, and data governance.

• Healthcare, Medicaid, Claims Processing, or Insurance domain experience.

• AWS or Cloudera certifications.

Key Responsibilities:

• Design and develop scalable PySpark data pipelines.

• Build and optimize ETL/ELT workflows using Cloudera CDP and Airflow.

• Implement Apache Iceberg-based data lakehouse solutions.

• Optimize distributed data processing and application performance.

• Support cloud migration, modernization, and production deployments.

• Collaborate with architects and engineering teams on enterprise data solutions.