Kafka Operations Administrator

Must Have Technical/Functional Skills

• Production-grade Apache Kafka operations experience, managing, maintaining and upgrading Kafka clusters in production environments with a focus on high availability, disaster recovery, fail-over and overall reliability.
• Proficiency in installing and configuring monitoring systems using Grafana (building dashboards), Prometheus, Splunk, JMX metrics.
• Automation and orchestration experience: Terraform, Ansible, Helm, Kubernetes (EKS/AKS/GKE).
• Strong Linux system administration experience, including troubleshooting, automation and scripting for efficient infrastructure management.
• Experience in Production Support (ITIL processes followed) and participating in 24x7 on-call rotations, documenting incidents/postmortems.
• Experience in supporting JVM tuning, Analysis, network and disk I/O diagnostics.
• Experience in TCP/IP, routing, switching and firewall configurations relevant to Kafka operations.

Good to Have

• Deep Kafka performance tuning and capacity planning experience.
• Knowledge of message delivery semantics and guarantees (at-least-once, exactly-once).
• Cloud-native security/compliance experience (IAM, VPC, KMS, Security Groups).
• Certifications: Confluent Certified Administrator, AWS/Azure/Google Cloud Platform certifications.
• Experience with Apache Kafka in KRaft mode, including set up, configuration, troubleshooting and cluster management.
• Containerization and container orchestration tools experience: Docker, Kubernetes.
• Experience with CI/CD pipelines and Git-based workflows.
• Experience building custom Kafka connect libraries and understanding of data serialization formats (e.g., Avro, JSON).
• Knowledge of networking concepts across on-prem VMs and cloud environments, ensuring seamless integration and communication between services.
• Strong understanding of topic management and security best practices for streaming platforms: TLS, ACLs, RBAC, encryption at rest/in transit.
• Kafka ecosystem tooling experience: Kafka Connect, Schema Registry.

Roles & Responsibilities

• Deploy, configure and manage Kafka clusters and related services to meet SLA requirements.
• Participate in 24x7 on-call rotation to respond to incidents, alerts, and escalations.
• Triage, diagnose, and remediate production incidents; coordinate with stakeholders, developers and infrastructure teams.
• Implement automation for provisioning, scaling, server/data backups, and disaster recovery.
• Maintain monitoring, alerting thresholds, dashboards, and Kafka ecosystem health.
• Harden Kafka deployments: configure TLS, ACLs, RBAC, encryption, and vulnerability remediation.
• Perform routine maintenance: Kafka ecosystem upgrades (controllers, brokers, connect, and schema registry), rolling restarts, etc.
• Create and maintain runbooks, runbook automation, and post-incident reports.
• Optimize performance and resource utilization; benchmark and tune clusters.
• Support Kafka Connect/Schema Registry service and troubleshoot connector issues.
• Contribute to CI/CD pipeline improvements for infrastructure and deployment automation.