Senior Big Data / Hadoop / Kafka / DevOps Administrator
Job Summary
Experienced AWS/Kafka Administrator responsible for managing, supporting, automating, and optimizing enterprise-scale Hadoop, Kafka, AWS cloud, and DevOps platforms.
The role focuses on ensuring high availability, performance, security, scalability, and operational excellence across big data ecosystems while collaborating with development,
architecture, and business teams.
Key Responsibilities
Kafka Administration with MSK Experience must have
Deploy, configure, and manage Apache Kafka clusters and AWS MSK environments.
Monitor broker health, partitions, replication factors, and consumer lag.
Perform capacity planning and cluster scaling activities.
Manage Kafka security using SSL, SASL, ACLs, and encryption standards.
Troubleshoot producer, consumer, and broker performance issues.
Support Kafka Connect, Schema Registry, Cruise Control, and MirrorMaker implementations.
AWS Cloud Administration
Manage cloud infrastructure services including EC2, S3, IAM, VPC, EBS, CloudWatch, CloudTrail, KMS, and AWS Glue.
Support AWS Managed Streaming for Kafka (MSK), EMR, Lambda, and Airflow environments.
Implement cloud security best practices and governance controls.
Perform infrastructure provisioning and automation using Infrastructure as Code (IaC).
Monitor cloud resource utilization and optimize operational costs.
DevOps & Automation
Design and maintain CI/CD pipelines using Jenkins, GitHub Actions, GitLab CI, or similar tools.
Automate infrastructure deployment using Terraform, CloudFormation, and Ansible.
Manage source control repositories and release processes.
Implement monitoring and alerting solutions using Prometheus, Grafana, Splunk, ELK, or CloudWatch.
Support containerization technologies such as Docker and Kubernetes.
Develop automation scripts using Python, Shell, or Bash.
Required Skills
Hadoop Administration (Cloudera/CDP)
Apache Kafka Administration
AWS Cloud Services
Linux (RHEL/Rocky Linux)
Shell Scripting and Python
Jenkins, Git, Ansible, Terraform
Docker and Kubernetes
Monitoring Tools (Grafana, Prometheus, Splunk)
Networking, Security, and High Availability Concepts
Performance Tuning and Capacity Planning
Preferred Qualifications
Bachelor s degree in Computer Science, Information Technology, or related field.
Experience with Cloudera CDP, AWS MSK, Airflow, and Spark.
AWS, Cloudera, Kafka, or Kubernetes certifications.
Experience supporting large-scale production environments handling petabyte-scale data workloads.
