Senior Middleware Engineer

Sr. Middleware Engineer – Kafka 
 , NJ/NY

Primary Responsibilities
1.    Architecture & Design
o    Architect, design, and implement Kafka-based solutions using Confluent Cloud and Confluent Platform, ensuring they are highly scalable, resilient, and future-proof.
o    Provide technical leadership in designing event-driven architectures that integrate with on-prem systems and multiple cloud environments (AWS, Azure, or Google Cloud Platform).
2.    Platform Management
o    Oversee administration and operational management of Confluent Platform components: Kafka brokers, Schema Registry, Kafka Connect, ksqlDB, and REST Proxy.
o    Develop and maintain Kafka producers, consumers, and streams applications to support real-time data streaming use cases.
3.    Deployment & Automation
o    Lead deployments and configurations of Kafka topics, partitions, replication strategies in both on-prem and cloud setups.
o    Automate provisioning, deployment, and maintenance tasks with Terraform, Chef, Ansible, Jenkins, or similar CI/CD tools.
4.    Monitoring & Troubleshooting
o    Implement robust monitoring, alerting, and observability frameworks using Splunk, Datadog, Prometheus, or similar tools for both Confluent Cloud and on-prem clusters.
o    Proactively troubleshoot Kafka clusters, diagnose performance issues, and conduct root cause analysis for complex, distributed environments.
5.    Performance & Capacity Planning
o    Conduct capacity planning and performance tuning to optimize Kafka clusters; ensure they can handle current and future data volumes.
o    Define and maintain SLA/SLI metrics to track latency, throughput, and downtime.
6.    Security & Compliance
o    Ensure secure configuration of all Kafka and Confluent components, implementing best practices for authentication (Kerberos/OAuth), encryption (SSL/TLS), and access control (RBAC).
o    Collaborate with InfoSec teams to stay compliant with internal and industry regulations (GDPR, SOC, PCI, etc.).
7.    Cross-Functional Collaboration
o    Work with DevOps, Cloud, Application, and Infrastructure teams to define and align business requirements for data streaming solutions.
o    Provide guidance and support during platform upgrades, expansions, and new feature rollouts.
8.    Continuous Improvement
o    Stay current with Confluent Platform releases and Kafka community innovations.
o    Drive continuous improvement by recommending new tools, frameworks, and processes to enhance reliability and developer productivity.
______________
Qualifications
1.    Technical Expertise
o    5+ years of hands-on experience with Kafka; at least 2+ years focused on Confluent Cloud and Confluent Platform.
o    Deep knowledge of Kafka Connect, Schema Registry, Control Center, ksqlDB, and other Confluent components.
o    Experience architecting and managing hybrid Kafka solutions in on-prem and cloud (AWS, Azure, Google Cloud Platform).
o    Advanced understanding of event-driven architecture and the real-time data integration ecosystem.
o    Strong programming/scripting skills (Java, Python, Scala) for Kafka-based application development and automation tasks.
2.    DevOps & Automation
o    Hands-on experience with Infrastructure as Code (Terraform, CloudFormation) for Kafka resource management in both cloud and on-prem.
o    Familiarity with Chef, Ansible, or similar configuration management tools to automate deployments.
o    Skilled in CI/CD pipelines (e.g., Jenkins) and version control (Git) for distributed systems.
3.    Monitoring & Reliability
o    Proven ability to monitor and troubleshoot large-scale, distributed Kafka environments using Splunk, Datadog, Prometheus, or similar tools.
o    Experience with performance tuning and incident management to minimize downtime and data loss.
4.    Security & Compliance
o    Expertise in securing Kafka deployments, including Kerberos and SSL configurations.
o    Understanding of IAM best practices, network security, encryption, and governance in hybrid environments.
5.    Leadership & Collaboration
o    Demonstrated experience leading platform upgrades, migrations, and architecture reviews.
o    Excellent communication skills, with ability to articulate complex technical concepts to diverse audiences (developers, architects, executives).
o    Comfortable collaborating with cross-functional teams—product owners, system engineers, security, and business stakeholders.
6.    Education & Preferred Experience
o    Bachelor’s or Master’s degree in Computer Science, Information Systems, or related field (or equivalent experience).
o    Experience with container orchestration (Docker/Kubernetes) is a plus.