Senior Data Engineer with Strong Kafka Streaming Exp - Atlanta, GA and Chicago, IL(Onsite)

Overview

On Site
Depends on Experience
Contract - Independent
Contract - W2
Contract - 12 month(s)
No Travel Required

Skills

JAVA
docker
data engineer
ETL
CLOUD
Kafka
API
Data Governance
Data Modeling
data pipelines
observability
real time streaming

Job Details

Role Name: Senior Data Engineer
Location: Atlanta, GA and Chicago, IL(Onsite)
Duration: 12 Months

Role Summary:
As a Senior Software Engineer, you’ll design and deliver real time data pipelines and event driven microservices. You’ll implement resilient streaming patterns, contribute to data governance and CI/CD automation, and collaborate closely with platform and product teams to scale our data services.

Responsibilities:
• Build reliable streaming applications using Confluent Kafka, Apache Flink, Hazelcast, Kafka Streams, Kafka Connect, and Schema Registry.
• Develop ETL/ELT pipelines for real time ingestion, transformation, and distribution; implement windowing, joins, and stateful processing.
• Implement distributed caching and in memory data grid integrations to reduce latency and improve throughput.
• Contribute to event gateway / event grid routing, schemas, topic design, ACLs, and dead letter strategies.
• Write clean, testable code for microservices (Java/Python), focusing on reliability, idempotency, and observability.
• Automate CI/CD pipelines, containerization (Docker), and deployments to Kubernetes.
• Participate in data governance: tagging, metadata updates, lineage capture, schema evolution, and data quality checks.
• Monitor production systems, perform performance tuning, troubleshoot backpressure/lag, and improve SLO attainment.
• Collaborate on design docs, code reviews, and cross team integrations.

Required Qualifications:
• 7+ years in software engineering, with 3+ years focused on real time streaming or event driven systems.
• Strong hands-on experience with Kafka (topics, partitions, consumer groups), Schema Registry, Kafka Connect, and either Flink or Kafka Streams or Hazelcast.
• Solid understanding of ETL/ELT concepts, event time vs. processing time, checkpointing, state management, and exactly once/at least once semantics.
• Proficiency with microservices (Java /Python), APIs (REST/gRPC), Avro/JSON/protobuf, and contract testing.
• Experience with Docker, Kubernetes, and CI/CD tools (GitHub Actions/Azure DevOps/Jenkins or similar).
• Familiarity with distributed caching (Redis, Hazelcast) and in memory data grids.
• Cloud experience in at least one cloud platform (Azure/AWS/Google Cloud Platform).
• Knowledge of observability (metrics, logs, traces) and resilience (retries, timeouts, DLQs, circuit breakers).
• Exposure to data governance, metadata catalogs, and lineage tooling; schema evolution and compatibility (backward/forward/full).
• Core competencies to include Problem Solving, Ownership, Code Quality, Operational Mindset, Collaboration, Continuous Improvement.

Nice to Have:
• CDC pipelines, Flink SQL, custom Kafka Connect connectors.
• Experience with feature flags, canary releases, and progressive delivery.
• Performance tuning: batch sizes, compression, serialization choices (Avro vs. Protobuf), and partitioning strategies.

Best Regards,

Vishal

Truth Lies in  Heart

 

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.