Senior Data Engineer

Project Overview
Responsible for managing critical Business-As-Usual (BAU) services that support enterprise-wide data operations.
These services include the development, maintenance, and monitoring of data pipelines, integrations, and reporting infrastructure that are essential for ongoing business functions.

Key responsibilities include:
1) Maintaining and troubleshooting ETL workflows (Pentaho, Databricks, ADF)
2) Supporting daily data loads and ensuring data availability for business reporting
3) Responding to ad-hoc requests from business users
4) Coordinating with DBAs and application teams for incident resolution
5) Performing enhancements to support evolving business data needs. These BAU services are essential for keeping business operations running smoothly and delivering timely insights across multiple departments

Job Functions & Responsibilities
ETL & Data Integration:
- Design, develop, and optimize ETL pipelines using Azure Databricks, ADF, and Pentaho to support enterprise data workflows.
- Implement and maintain data movement, transformation, and integration across multiple systems.
- Ensure seamless data exchange between cloud, on-prem, and hybrid environments.
- Work with Globalscape FTP for secure file transfers and automation. API Development and Integration:
- Develop, consume, and integrate RESTful and SOAP APIs to facilitate data .
- Work with API gateways and authentication methods such OAuth, JWT, certificate, and API keys.
- Implement and optimize API-based data extractions and real-time data integrations

Data Quality & Governance:
- Implement data validation, cleansing, and enrichment techniques.
- Develop and execute data reconciliation processes to ensure accuracy and completeness.
- Adhere to data governance policies and security compliance standards.
BAU Support & Performance Optimization:
- Troubleshoot and resolve ETL failures, data load issues, and performance bottlenecks**.
- Optimize SQL stored procedures and complex queries for better performance.
- Support ongoing enhancements and provide operational support for existing data pipelines.
Collaboration & Documentation:
- Work closely with Data Analysts, Business Analysts, and stakeholders to understand data needs.
- Document ETL processes, data mappings, and workflows for maintainability and knowledge sharing.
- Provide guidance and best practices to ensure the scalability and efficiency of data solutions.

Required Skills & Experience:
- 7+ years of experience in ETL development, data integration, and SQL scripting.
- Strong expertise in Azure Databricks, ADF (Azure Data Factory), and Pentaho.
- Experience handling secure file transfers using Globalscape FTP.
- Hands-on experience in developing and consuming APIs (REST/SOAP).
- Experience working with API security protocols (Oauth, JWT, API Keys, etc.,).
- Proficiency in SQL, stored procedures, performance tuning, and query optimization.
- Understanding of data modeling, data warehousing, and data governance best practices.
- Hands-on experience with cloud-based data platforms (Azure/AWS) is a plus.
- Strong problem-solving skills, troubleshooting abilities, and ability to work independently.
- Excellent communication skills and ability to work in a fast-paced environment. Preferred Qualifications:
- Experience working in large-scale enterprise data integration projects.
- Knowledge of Python, PySpark for big data processing.
- Familiarity with CI/CD for data pipelines (Azure DevOps, GitHub Actions, etc.).

Education & Certifications
Bachelor's or Master's degree in a relevant field like Computer science , Data Engineering or related technical field.

Nice to have below certifications:
a) Databricks certified Data Engineer
b) Azure Data Engineer associate