Kafka Consultant

Job Summary

We are seeking an experienced Kafka Consultant to design, implement, optimize, and support enterprise-grade event-driven data streaming solutions using Apache Kafka. The ideal candidate will have strong expertise in Kafka architecture, distributed systems, cloud platforms, and real-time data integration. This role involves collaborating with architects, developers, DevOps engineers, and business stakeholders to deliver scalable, secure, and highly available streaming platforms.

Key Responsibilities

• Design, develop, and implement Apache Kafka-based event streaming solutions.
• Install, configure, administer, and optimize Kafka clusters for high availability and performance.
• Develop producers, consumers, and stream processing applications using Kafka APIs.
• Implement Kafka Connect, Kafka Streams, ksqlDB, and Schema Registry.
• Configure and manage security features including SSL/TLS, SASL, ACLs, and RBAC.
• Monitor Kafka clusters and troubleshoot production issues related to performance, latency, and reliability.
• Perform capacity planning, cluster scaling, partition management, and replication optimization.
• Integrate Kafka with databases, cloud services, REST APIs, and enterprise applications.
• Automate deployment and infrastructure using CI/CD pipelines and Infrastructure as Code (Terraform/Ansible).
• Collaborate with development teams to implement event-driven architectures and messaging best practices.
• Document architecture, operational procedures, and technical solutions.
• Provide production support and participate in on-call rotations when required.
• Mentor development teams on Kafka best practices and streaming design patterns.

Required Qualifications

• Bachelor''s degree in Computer Science, Information Technology, Engineering, or related field.
• 10+ years of experience in distributed systems and middleware technologies.
• 5-8 years of hands-on experience with Apache Kafka administration and development.
• Strong knowledge of Kafka internals including brokers, partitions, replication, consumer groups, and offsets.
• Experience with Kafka Connect, Kafka Streams, Schema Registry, and ksqlDB.
• Proficiency in Java, Scala, or Python.
• Experience with Linux system administration.
• Knowledge of Docker and Kubernetes.
• Familiarity with CI/CD tools such as Jenkins, GitHub Actions, or GitLab CI.
• Experience with cloud platforms (AWS, Azure, or Google Cloud Platform).

Preferred Qualifications

• Experience with Confluent Platform.
• Knowledge of Apache Spark, Apache Flink, or Apache NiFi.
• Experience with monitoring tools such as Prometheus, Grafana, Splunk, or ELK Stack.
• Familiarity with microservices architecture and event-driven design patterns.
• Experience with Infrastructure as Code tools such as Terraform or Ansible.
• Confluent Certified Developer or Administrator certification is a plus.