PySpark lead developer

Hybrid in Irving, TX, US • Posted 17 hours ago • Updated 17 hours ago
Full Time
Hybrid
$100,000 - $120,000/yr
Fitment

Dice Job Match Score™

🫥 Flibbertigibetting...

Job Details

Skills

  • PySpark
  • ETL
  • Sql
  • Data Engineer

Summary

Responsibilities: Development and Maintain Data Pipelines: Design, implement, and optimize end-to-end ETL/ELT pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data. Utilize Python and Pyspark: Write efficient, scalable and maintainable code in Python and leverage Pyspark for large-scale data processing in distributed computing environments. Also be able to review existing code and identify areas of improvement. Ensure Data Quality and Integrity: Implement data validation, cleansing, transformation and reconciliation processes to ensure data accuracy and consistency throughout the data lifecycle. Collaborate with Stakeholders: Work closely with IT teams and business stakeholders to gather data requirements and translate them to technical solutions. Troubleshoot and Optimize: Monitor job performance, troubleshoot complex data issues and fine-tune for performance and scalability. Adhere to Best Practices: Participate in code reviews, establish coding standards, and implement CI/CD pipelines for automated testing and deployment. Skills: Strong hands-on coding proficiency in Python, Pyspark and SQL (Microsoft SQL Server preferred) Experience with big data frameworks (Hadoop, Spark). Experience with cloud platforms ( AWS, Azure or Google Cloud Platform) Experience with Code versioning tools ( Bitbucket, Github ) Experience with CI/CD and setting up pipelines. Solid understanding of database design principles, data modelling, schemas and data warehousing solutions. Excellent problem-solving and analytical skills to troubleshoot complex data issues independently.

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
  • Dice Id: hexaware
  • Position Id: 9020160
  • Posted 17 hours ago
Contact the job poster
MK

Manish Kumar

Recruiter @ Hexaware Technologies, Inc
Create job alert
Set job alertNever miss an opportunity! Create an alert based on the job you applied for.

Similar Jobs

Irving, Texas

Today

Easy Apply

Full-time

Depends on Experience

Irving, Texas

15d ago

Easy Apply

Full-time

120,000 - 140,000

Irving, Texas

17d ago

Easy Apply

Full-time

Depends on Experience

Irving, Texas

Today

Full-time

Search all similar jobs