Data Engineer / ETL Developer

Introduction:

As a Data Engineer / ETL Developer, you will play a crucial role in our team by developing and maintaining ETL pipelines, optimizing data workflows, and ensuring the smooth operation of our data infrastructure. You will work closely with cross-functional teams to support various projects and initiatives that require efficient data processing and analysis.

Responsibilities:

• Develop and optimize ETL pipelines using Apache NiFi, PySpark, and AWS Glue.
• Write and optimize complex SQL queries for data extraction, transformation, and loading.
• Work with MySQL, PostgreSQL, and AWS DocumentDB to manage and manipulate data efficiently.
• Utilize Apache Kafka or similar streaming platforms for real-time data processing.
• Provide development and production support for various projects and initiatives.
• Troubleshoot production issues, perform root cause analysis, and implement necessary fixes.
• Coordinate and execute System Integration Testing, Regression Testing, and data validation processes.

Requirements:

Required Skills:

• At least 3 years of hands-on experience in Apache Kafka or similar streaming platforms.
• Strong proficiency in Apache NiFi, PySpark, and AWS Glue for ETL development.
• Expertise in writing and optimizing complex SQL queries, performance tuning, and database management (MySQL, PostgreSQL, AWS DocumentDB).

Preferred Skills:

• Experience with other data processing technologies such as Apache Flink or Apache Beam.
• Familiarity with cloud platforms such as AWS, Azure, or Google Cloud Platform.
• Knowledge of data warehousing concepts and tools.