Kafka Administrator with Google Cloud Platform and Terraform

Role: Kafka Administrator with Google Cloud Platform and Terraform

Location: Remote

Key Skills - Must Have

Confluent Kafka

Google Cloud Engineer

Network Security

Terraform

Strong Customer Communication Skills

Key Skills - Good to Have

GitLab CI/CD Pipelines.

Docker, Kubernetes.

Experience:

10+ years of experience in infrastructure, with 3+ years as a Kafka Administrator or in a similar role managing Kafka clusters.

Proven experience as a Kafka Administrator or in a similar role managing Kafka clusters.

Strong understanding of distributed systems, data streaming, and event-driven architectures.

Experience with Linux/Unix operating systems and shell scripting.

Proficiency in Google Cloud Platform and containerization technologies (Docker, Kubernetes) is a plus.

Experience with network security principles and practices.

Proficiency in Terraform for infrastructure as code.

Experience with GitLab CI/CD pipelines is a plus.

Strong customer communication skills to help expand and secure more work with clients.

Job Responsibilities

Key Responsibilities:

1. Cluster Management:

o Install, configure, and maintain Kafka clusters across various environments (development, testing, production).

o Perform upgrades and patching of Kafka and related components (e.g., Zookeeper).

o Ensure optimal performance and reliability of Kafka clusters.

2. Monitoring and Troubleshooting:

o Monitor Kafka cluster health and performance using tools like Prometheus, Grafana, or proprietary monitoring solutions.

o Diagnose and resolve issues related to Kafka brokers, topics, partitions, and consumers.

o Implement proactive measures to prevent potential issues.

3. Security and Compliance:

o Implement and manage security protocols for Kafka, including SSL/TLS encryption, Kerberos authentication, and access control policies.

o Ensure compliance with organizational and industry standards for data security and privacy.

o Apply network security principles to protect Kafka infrastructure.

4. Capacity Planning and Scalability:

o Perform capacity planning to ensure the Kafka infrastructure can handle current and future workloads.

o Optimize Kafka configurations for performance and scalability based on application requirements.

5. Backup and Recovery:

o Develop and maintain disaster recovery plans for Kafka environments.

o Implement and test backup and restore procedures to ensure data integrity and availability.

6. Collaboration and Support:

o Work closely with development teams to understand Kafka usage patterns and provide guidance on best practices.

o Provide support for Kafka-related issues, including on-call support as needed.

o Document Kafka infrastructure, configurations, and operational procedures.

7. Automation and Scripting:

o Develop automation scripts for routine tasks such as cluster provisioning, monitoring, and maintenance using tools like Ansible, Puppet, or custom scripts.

o Implement CI/CD pipelines for Kafka-related deployments and updates.

8. Client Communication and Reporting:

o Maintain regular communication with clients to provide updates and gather feedback.

o Prepare and present weekly and monthly status reports to stakeholders.

o Present proof-of-concept (POC) solutions and designs to clients and internal teams.

Department/Project Description

The Kafka Administrator is responsible for managing, maintaining, and monitoring Apache Kafka clusters within an organization. This role involves ensuring high availability, performance, and security of Kafka environments. The Kafka Administrator collaborates with development, infrastructure, and operations teams to support Kafka-based applications, troubleshoot issues, and optimize the platform for scalability and reliability.

Skill Category

DevOps

Key Skills - Must Have

Confluent Kafka

Google Cloud Engineer

Network Security

Terraform

Strong Customer Communication Skills

Key Skills - Good to Have

GitLab CI/CD Pipelines.

Docker, Kubernetes.