Sr. Middleware Engineer Kafka

Overview

On Site
Full Time

Skills

Java
python
DEVOPS
DOCKERS
Scala
Kafka
kubernetes
Jenkins
Apache Kafka
ansible
Terraform
Git
Chef

Job Details

Job Title: Sr. Middleware Engineer Kafka
Location: New York (NY) or New Jersey (NJ)
Primary Responsibilities
Architecture & Design:
  • Architect, design, and implement Kafka-based solutions using Confluent Cloud and Platform.
  • Provide leadership in event-driven architecture across on-prem and cloud (AWS, Azure, Google Cloud Platform).
Platform Management:
  • Administer and manage Kafka components: brokers, Schema Registry, Kafka Connect, ksqlDB, REST Proxy.
  • Develop Kafka producers, consumers, and streaming applications.
Deployment & Automation:
  • Lead Kafka topic deployments and configurations (partitions, replication).
  • Automate deployment and maintenance with Terraform, Chef, Ansible, Jenkins, etc.
Monitoring & Troubleshooting:
  • Set up observability tools like Splunk, Datadog, Prometheus.
  • Troubleshoot performance issues and ensure cluster health.
Performance & Capacity Planning:
  • Plan for capacity, tune performance, and define SLA/SLI metrics.
Security & Compliance:
  • Ensure secure Kafka setup (Kerberos/OAuth, SSL/TLS, RBAC).
  • Align with compliance standards (GDPR, SOC, PCI).
Collaboration & Improvement
  • Partner with DevOps, Cloud, App, and Infra teams.
  • Stay current on Kafka/Confluent innovations; recommend improvements.
Qualifications
Technical Expertise:
  • 5+ years with Apache Kafka; 2+ years with Confluent Cloud/Platform.
  • Expertise in Kafka Connect, Schema Registry, Control Center, ksqlDB.
  • Experience with hybrid on-prem/cloud Kafka architectures.
  • Proficient in Java, Python, or Scala.
DevOps & Automation:
  • Skilled with Terraform, CloudFormation, Jenkins, Git.
  • Familiar with Chef, Ansible, and CI/CD practices.
Monitoring & Reliability:
  • Experience with Splunk, Datadog, Prometheus for monitoring.
  • Performance tuning and incident management skills.
Security & Compliance:
  • Strong with Kerberos, SSL, IAM, encryption, and governance.
Leadership & Collaboration:
  • Led platform upgrades and migrations.
  • Clear communicator across technical and business teams.
Education & Preferred:
  • Bachelor's/Master's in CS, IS, or related field.
  • Docker/Kubernetes experience is a plus.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.