Job Title: Senior Data Pipeline & Ingestion Engineer
Experience: 5+ Years
Employment Type: Full-Time
Level: Mid / Senior / Lead
About the Role
We are looking for an experienced Data Pipeline & Ingestion Engineer to design and build large-scale batch and streaming data pipelines.
The role focuses on Kafka, Python/Java, SQL, ETL, data ingestion, Medallion/Lakehouse architecture, CDC, data quality, reconciliation, and entity resolution/MDM.
You will work on the core data platform responsible for ingesting data from multiple legacy and enterprise systems, transforming it into standardized data models, and ensuring data quality and reconciliation before publishing.
This reflects the uploaded ODL requirement, which specifically emphasizes Kafka, batch/stream ingestion, medallion layers, reconciliation, identity matching, source-to-canonical mappings, and data contracts.
Responsibilities
- Design and build scalable batch and streaming data pipelines.
- Develop data ingestion solutions using Kafka.
- Implement consumer/producer, replay, DLQ, and idempotent processing patterns.
- Build and maintain Bronze, Silver, and Gold Medallion data layers.
- Implement CDC, watermark, checkpoint, retry, and batch-to-stream handoff patterns.
- Develop strong data-quality validation and reconciliation processes.
- Implement quarantine, reprocessing, and data-quality scoring.
- Perform count, record-level, and financial reconciliation.
- Develop identity matching and entity resolution / MDM solutions.
- Implement record matching, deduplication, golden records, and survivorship rules.
- Build and maintain source-to-canonical data mappings and crosswalks.
- Work with structured and semi-structured enterprise datasets.
- Implement schema validation and schema evolution.
- Support production monitoring, troubleshooting, and performance optimization.
Mandatory Skills
| Skill | Requirement |
|---|
| Data Engineering / ETL | Mandatory |
| Kafka | Mandatory – Strong hands-on |
| SQL | Mandatory |
| Java OR Python | Mandatory |
| Batch + Streaming Pipelines | Mandatory |
| CDC | Mandatory |
| Medallion / Lakehouse | Mandatory |
| Data Quality | Mandatory |
| Data Reconciliation | Mandatory |
| Entity Resolution / MDM | Strongly Preferred |
| Data Mapping / Crosswalks | Strongly Preferred |
| Git / YAML / JSON | Preferred |
| Avro / Protobuf / Schema Registry | Preferred |
Good to Have
- Informatica MDM
- Reltio
- Probabilistic record matching
- Golden-record implementation
- Avro / Protobuf
- Schema Registry
- Mainframe or legacy RDBMS ingestion
- Financial reconciliation
- Healthcare / Benefits domain experience
Never miss an opportunity! Create an alert based on the job you applied for.
