Kafka Administrator

Job Title: Kafka Administrator
Location: New York, NY (Hybrid – 3 days onsite)

Position Overview

We are looking for an experienced Kafka Administrator to manage and support our Kafka ecosystem. This role will focus on ensuring high availability, performance, and security of Kafka clusters while collaborating with cross-functional teams to support real-time data streaming and integration needs.

Key Responsibilities

• Kafka Cluster Management:
  Install, configure, and maintain Kafka clusters, ensuring high availability and implementing disaster recovery strategies.
• Performance Optimization:
  Monitor cluster performance, identify bottlenecks, and fine-tune configurations to optimize throughput and latency.
• Security Management:
  Implement and manage Kafka security including authentication, authorization, and encryption.
• Monitoring & Alerting:
  Set up and maintain monitoring tools and alerting systems to proactively detect and resolve issues.
• Troubleshooting:
  Diagnose and resolve Kafka-related issues including performance degradation, connectivity problems, and data inconsistencies.
• Integration Support:
  Work closely with development and data teams to integrate Kafka with various applications and data pipelines.
• Documentation:
  Create and maintain technical documentation for configurations, processes, and operational procedures.
• Automation:
  Develop scripts and automate routine operational tasks to improve efficiency and reliability.
• Capacity Planning:
  Analyze current usage and forecast future needs to ensure scalability and optimal resource utilization.

Required Skills & Qualifications

• Strong experience with Apache Kafka administration and cluster management
• Hands-on experience with Kafka tools and ecosystem (Zookeeper, Kafka Connect, Schema Registry)
• Experience with monitoring tools (Prometheus, Grafana, or similar)
• Solid understanding of distributed systems and data streaming concepts
• Experience with scripting (Shell, Python, or similar)
• Knowledge of security protocols (SSL, SASL, ACLs)
• Strong troubleshooting and problem-solving skills

Preferred Qualifications

• Experience with cloud platforms (AWS, Azure, or Google Cloud Platform)
• Familiarity with containerization technologies (Docker, Kubernetes)
• Experience with CI/CD pipelines and automation tools