Big Data Engineer/Big Data Developer

We are seeking a Big Data Engineer with strong Python and Apache Spark experience to design and build scalable data processing systems. The candidate will work with large datasets, develop reliable data pipelines, and collaborate with analytics and data science teams to support data-driven business decisions.

Key Responsibilities

• Design and implement scalable data processing pipelines using Python and Apache Spark (PySpark).

• Develop and maintain ETL/ELT workflows to process large volumes of data efficiently.

• Collaborate with data scientists, analysts, and engineering teams to improve data pipelines and workflows.

• Perform data cleansing, transformation, and aggregation for analytics and reporting requirements.

• Ensure data consistency, quality, and reliability across multiple data sources.

• Troubleshoot and optimize Spark jobs and data processing applications to improve performance.

• Integrate big data solutions with existing enterprise systems and databases.

• Monitor data pipelines in production and ensure system stability and reliability.

• Follow coding standards, documentation practices, and participate in peer code reviews.

Roles & Responsibilities

• Build distributed data processing applications using Apache Spark.

• Write efficient and reusable Python code for data engineering and pipeline development.

• Analyze large datasets to identify patterns, insights, and trends for business use cases.

• Work with stakeholders to gather requirements and translate them into technical data solutions.

• Provide guidance and mentorship to junior developers within the team.

• Support deployment, monitoring, and maintenance of production data pipelines.

Required Skills

• Strong programming experience in Python.

• Hands-on experience with Apache Spark / PySpark for big data processing.

• Experience building and maintaining data pipelines and ETL processes.

• Strong SQL skills and data processing knowledge.

• Understanding of data modeling and distributed data processing frameworks.