PySpark / Python Data Engineer Tech Lead role

Hi,

Role: PySpark / Python Data Engineer Tech Lead role

Location Remote

Duration: 6+ Months

Job Summary:

We are seeking a seasoned professional with expertise in building data engineering and analytics solutions within AWS ecosystems. The ideal candidate should have deep experience in PySpark, Python, and endtoend data pipeline development, including job orchestration, workflow design, and data mapping. The role requires the ability to translate complex business logic, stored procedures, and SQL triggers into scalable PySpark implementations. Experience with data streaming on Spark clusters and API design is highly desirable. Knowledge of Palantir Foundry is a strong plus.

Details:

Minimum Qualifications: MS or equivalent experience in Computer Science, MIS, or related technical fields; 10 15+ years of overall experience, with 5+ years in data engineering/ETL ecosystems using PySpark, Python, and Java.

Key Responsibilities:

- Translate business requirements into technical solutions using PySpark and Python frameworks.

- Lead data engineering initiatives for complex analytics challenges.

- Plan and execute tasks, track progress, and document work following best practices.

- Identify and implement process improvements, including scalable infrastructure design and workflow automation.

- Participate in Agile/Scrum ceremonies.

- Provide technical guidance to team members across functional and technical domains.

- Build infrastructure for largescale data access and ensure data quality/metadata management.

- Collaborate with leadership to strengthen datadriven decisionmaking.

Required Skills:

- Strong expertise in PySpark and Python.

- Experience with Pandas, APIs, and Spark Streaming.

- Solid understanding of database design fundamentals.

- Familiarity with CI/CD tools and infrastructureascode frameworks.

- Experience writing productiongrade code, including unit/integration tests and schema validations.

- Knowledge of Palantir Foundry (Ontology modeling, API configuration, Foundry Typescript) and exposure to Power BI or Tableau are significant advantages.