Senior AWS Data Engineer with Collibra

Job Title: Senior AWS Data Engineer with Collibra

JOB DESCRIPTION :

The role will focus on modernizing legacy batch processing workloads, migrating and rewriting existing jobs into AWS Batch, enhancing cloud-based data ingestion capabilities, and improving data quality, lineage, inventory, resiliency, and operational support processes.

The ideal candidate will have strong hands-on experience in Collibra and Atacama.

The ideal candidate will have strong hands-on experience in data engineering, batch modernization, cloud-based data processing, SQL development, job scheduling, and enterprise data platform delivery. This role requires the ability to work across legacy and modern technology environments, collaborate with business and technology stakeholders, and deliver reliable, scalable, secure, and audit-ready data solutions.

Experience Level: 9+ years of relevant data engineering experience, preferably in financial services, investment management, banking, or other highly regulated enterprise environments.

Key Responsibilities

• Analyze legacy batch jobs, scripts, database procedures, and data workflows to understand current-state processing, dependencies, business logic, scheduling requirements, and operational constraints.
• Rewrite and migrate legacy batch workloads into AWS Batch using modern engineering practices while preserving business functionality and meeting existing service-level agreements.
• Develop, enhance, and maintain data pipelines and batch processing solutions using Java Spring Batch, Python, SQL, shell scripting, and AWS services.
• Support integration between on-premises legacy systems and AWS-based target platforms, ensuring reliable data movement, synchronization, monitoring, and operational continuity.
• Build and optimize data ingestion workflows to support the Rapid Data Ingestion platform, including file intake, validation, transformation, inventory tracking, and downstream consumption.
• Implement data quality checks to detect changes in vendor file formats, data integrity issues, missing or incomplete data, and other anomalies that may impact critical business processes.
• Enable and support data lineage capabilities by capturing source-to-target data movement, transformations, metadata, and audit-relevant processing details.
• Support disaster recovery implementation for the Rapid Data Ingestion platform in AWS to improve resiliency and continuity for critical and audit-sensitive processes.
• Develop and maintain job scheduling and orchestration processes using Control-M or equivalent scheduling tools.
• Implement secure engineering practices, including application-specific database access, elimination of shared credentials, and enforcement of least-privilege access principles.
• Perform performance tuning to ensure rewritten jobs meet or exceed current batch cycle expectations, including completion within strict operational SLAs.
• Participate in data validation, reconciliation, defect triage, testing, production deployment, and warranty support activities.
• Collaborate with data analysts, systems analysts, architects, application teams, database teams, governance teams, and business stakeholders to ensure successful delivery.
• Create and maintain technical documentation covering design, mappings, dependencies, lineage, quality rules, operational procedures, and deployment details.

Required Technical Skills:

• Strong experience with Java Spring Batch for batch job development, migration, and modernization.
• Hands-on experience with AWS services, especially AWS Batch and Amazon S3.
• Strong SQL development experience, including Oracle SQL and PL/SQL concepts.
• Experience developing data engineering solutions using Python.
• Strong shell scripting experience using Bash, KornShell, or similar Unix/Linux scripting languages.
• Experience with Control-M or equivalent enterprise job scheduling and orchestration tools.
• Experience working with legacy batch processing environments and modernizing on-premises batch workloads to cloud-based platforms.
• Ability to analyze and convert legacy scripts, SQL loaders, stored procedures, and batch workflows into modern, maintainable data processing jobs.
• Experience with data ingestion, ETL/ELT development, data validation, reconciliation, and production support.
• Understanding of secure access patterns, credential management, least-privilege controls, and audit-focused data processing.
• Experience with CI/CD, code deployment, version control, and standard software engineering practices.

Education: Bachelor s degree in Computer Science, Information Technology, Engineering, Data Engineering, or a related field, or equivalent practical experience.

Certifications: AWS certification, Snowflake certification, or relevant cloud/data engineering certification is preferred but not mandatory.

Expected Deliverables:

• Modernized and rewritten batch jobs deployed to AWS Batch.
• Enhanced Rapid Data Ingestion platform capabilities.
• Data quality checks, validation routines, and exception handling processes.
• Data lineage and metadata capture support.
• File inventory management capabilities for incoming data feeds.
• Disaster recovery support for AWS-based ingestion processes.
• Secure application-specific access patterns and reduced credential-sharing risk.
• Control-M or equivalent job schedules and operational workflows.
• Technical documentation, deployment notes, runbooks, and support procedures.
• Data validation, reconciliation, performance tuning, and production support outcomes.