Title: Enterprise Data Lake (EDL) Technical Lead
Location: Remote
Duration: 12 Months contract
12+ years of experience
Must have: Strong Apache Kafka and Apache Flink expertise, Strong CDC experience - Integration, Reconciliation, and Recovery, Hands-on Apache Iceberg/Delta Table experience, Strong Databricks and Apache Spark experience, Working experience deploying and troubleshooting containerized data services on Kubernetes and Java experience for stream-processing applications and connector extensions.
JD:
Principal / Lead Data Engineer
Data Platform and Knowledge Engineering
Role Summary
We are looking for a hands-on Principal / Lead Data Engineer to build and evolve the data platform. You will design and deliver batch and streaming pipelines using Kafka, Flink, CDC, Apache Iceberg, and Databricks, making trusted data available for applications, analytics, and AI.
This role combines strong software engineering with deep data-engineering expertise. You will own capabilities across ingestion, stream processing, lakehouse storage, transformation, and serving—including deployment, monitoring, troubleshooting, and recovery. You will guide architectural decisions, mentor engineers, and establish reusable patterns across teams.
What You Will Do
· Build streaming data pipelines: Design and implement event-driven pipelines using Apache Kafka and Apache Flink, including stateful processing, enrichment, joins, and aggregations.
· Own CDC integration: Capture changes from operational databases using tools such as Qlik Replicate and Debezium. Handle initial snapshots, incremental updates, deletes, schema changes, and recovery.
· Develop the lakehouse: Build and maintain Apache Iceberg and Delta Lake datasets, defining table structure, partitioning, schema evolution, maintenance, and retention strategies.
· Engineer on Databricks: Develop production pipelines using Python, SQL, Spark, and Databricks for ingestion, transformation, reconciliation, and curated data delivery.
· Ensure data correctness: Design for duplicate and late-arriving events, event ordering, idempotency, replay, and backfills. Validate processing guarantees across the complete pipeline.
· Design reusable data models: Translate source-system structures and business requirements into models supporting historical analysis, reliable joins, and application consumption.
· Deliver governed data: Establish data quality, metadata, lineage, ownership, and access controls, including tenant isolation and protection of sensitive data.
· Automate delivery: Build automated tests, CI/CD pipelines, infrastructure as code, and repeatable deployments across environments.
· Operate production systems: Monitor freshness, completeness, lag, throughput, and failures. Troubleshoot issues across connectors, brokers, processing jobs, and storage.
· Optimize performance and cost: Tune streaming and batch workloads, resource allocation, file sizes, partitioning, and query performance.
· Provide technical leadership: Lead design and code reviews, mentor engineers, and establish practical standards and reusable implementation patterns.
What You Bring
· 12+ years of relevant software or data-engineering experience, including technical leadership for complex production data systems.
· Strong Apache Kafka expertise, including topic and partition design, consumer groups, offsets, message schemas, delivery semantics, and operational troubleshooting.
· Hands-on Apache Flink experience, including stateful stream processing, event time, watermarks, checkpoints, savepoints, recovery, and performance tuning.
· Strong CDC experience, including database change capture, snapshot-to-stream transitions, updates and deletes, schema evolution, reconciliation, and replay.
· Hands-on Apache Iceberg/Delta Table experience, including table design, schema and partition evolution, snapshots, compaction, and integration with processing engines and catalogs.
· Strong Databricks and Apache Spark experience, including production pipeline development, orchestration, distributed processing, troubleshooting, and optimization.
· Strong Python and SQL skills, with experience building maintainable, tested code beyond exploratory notebooks.
· Experience building data platforms on Azure, including storage, identity, networking, and secure connectivity.
· Java experience for stream-processing applications and connector extensions.
· Experience implementing automated data-quality checks, source-to-target reconciliation, lineage, and access controls.
· Practical experience with CI/CD, automated testing, and infrastructure as code for data workloads.
· Working experience deploying and troubleshooting containerized data services on Kubernetes.
· Ability to explain technical trade-offs, collaborate with domain experts, and guide engineers through implementation and production support.
Nice to Have
· Experience with Confluent Cloud, Schema Registry, Kafka Connect, and Tableflow.
· Experience with Qlik Replicate, Debezium, or custom connector development.
· Experience with Databricks Lakeflow, Unity Catalog, Delta Lake, and Delta Sharing.
· Familiarity with DataHub or similar metadata and lineage platforms.
· Experience serving curated data through PostgreSQL, Databricks Lakebase, or APIs.
· Experience preparing governed data for AI applications, semantic models, or entity resolution.
