Senior Data Engineer - Remote

Role : Senior Data Engineer  

Duration : 06+ Months

Remote

Skills : Hadoop, Spark, Kafka, DataStage

Domain : Banking is Must

Key Responsibilities:

• Build and support real-time and batch-based data pipelines using Big Data distributed systems and streaming technologies.
• Design, develop, test, and maintain scalable ETL and ELT pipelines for processing large volumes of structured and unstructured data.
• Develop data ingestion, transformation, and orchestration workflows using ETL tools such as IBM DataStage and modern scheduling/orchestration platforms.
• Write complex SQL queries, stored procedures, and optimization logic for high-performance data processing.
• Work extensively in UNIX/Linux environments for scripting, job automation, file handling, monitoring, and troubleshooting.
• Develop and maintain applications using programming languages such as Python and Java for automation, data processing, and integration tasks.
• Monitor production data pipelines, troubleshoot failures, perform root cause analysis, and provide production support within SLA timelines.
• Implement best practices for data quality, validation, reconciliation, logging, monitoring, and operational support.
• Work with orchestration and scheduling tools such as Airflow, Control-M, Autosys, or equivalent workflow automation platforms.
• Support cloud and Big Data initiatives involving Hadoop, Spark, Kafka, distributed processing systems, and real-time streaming frameworks.
• Participate in code reviews, technical design discussions, and Agile/Scrum ceremonies.

Required Qualifications:

• Must have 14+ years work experience in successful delivery of complex data related projects end to end.
• Experience with Big Data technologies such as Hadoop, Spark, Kafka, Hive, or distributed computing systems.
• Strong hands-on experience in ETL development and enterprise data integration projects.
• Expertise in IBM DataStage and data warehousing concepts.
• Strong SQL development and query optimization experience.
• Proficiency in UNIX/Linux shell scripting and system-level operations.
• Strong programming experience in Python and/or Java.
• Knowledge of real-time data pipeline development and streaming architectures.
• Experience with scheduling and orchestration tools such as Airflow, Autosys, Control-M, or equivalent.
• Experience supporting production environments including incident management, debugging, monitoring, and performance tuning.
• Strong analytical, troubleshooting, and communication skills.