Sr. Kafka Lead/Architect @ Remote

Title: Sr. Kafka Lead/Architect

Location: Remote

Duration: 6 – 12+ months

Job Summary:

We are seeking a Senior Kafka Consultant with deep expertise in event-streaming architecture, distributed systems, and real-time data engineering, combined with a proven track record of delivering production-grade platforms for Banking and Financial Services clients. This is a fully remote, hands-on, client-facing consulting engagement.

Key Responsibilities

• Lead Streaming & Real-Time Data Initiatives Collaborate with senior client stakeholders and strategic partners across Banking and Financial Services to architect, deploy, and scale Apache Kafka / Confluent Platform clusters for high-throughput, low-latency, fault-tolerant workloads. Design event-driven and streaming data architectures (event sourcing, CQRS, CDC, real-time ETL) aligned to banking use cases such as payments, fraud detection, AML, risk, and regulatory reporting. Conduct assessments, workshops, and capability presentations; respond to technical RFP components and shape streaming and real-time transformation programs.
• Architect Scalable Streaming Infrastructure Implement and optimize Kafka Streams, ksqlDB, Kafka Connect, and Schema Registry, including exactly-once semantics where required. Design reusable frameworks for data ingestion, transformation, and movement using Kafka and complementary streaming tools (Spark Streaming, Apache Flink). Advise on managed versus self-managed trade-offs across Confluent Cloud, Amazon MSK, and self-hosted deployments, and design multi-region / DR architectures (MirrorMaker 2, Cluster Linking).
• Drive Data Governance, Security, and Reliability Establish schema evolution, data contract standards, and governance across producers and consumers. Lead security and compliance hardening: encryption in transit and at rest, mTLS, RBAC/ACLs, audit logging, and alignment with financial regulatory and data-residency requirements. Build observability and SRE practices around the platform, including monitoring, alerting, capacity planning, and disaster recovery.
• Engage with Senior Tech Stakeholders Understand enterprise goals, identify real-time and automation opportunities, and propose cutting-edge streaming solutions that integrate with broader data and AI/ML workflows. Troubleshoot production incidents — consumer lag, rebalancing, partition skew, broker performance — and drive root-cause resolution. Mentor client engineering teams and document architecture, runbooks, and best practices for long-term ownership.

Skills & Attributes for Success

• Deep Kafka & Distributed Systems Knowledge - 7+ years in data engineering / distributed systems, with 4+ years of deep, production Apache Kafka experience. Expert knowledge of Kafka internals: partitioning, replication, consumer groups, ISR, offset management, and delivery semantics. Strong hands-on experience across the ecosystem: Kafka Connect, Kafka Streams, ksqlDB, and Schema Registry (Avro / Protobuf / JSON Schema).
• Cloud and Big Data Expertise - Proven architecture experience across AWS, Azure, or Google Cloud Platform, with proficiency in containers (Docker) and orchestration (Kubernetes). Experience integrating Kafka with downstream systems such as databases, data lakes/warehouses, and microservices. Familiarity with the broader streaming and big data landscape, including Spark Streaming and Apache Flink.
• Real-Time Engineering and Performance Proven experience operating Kafka at scale in production, including performance tuning, benchmarking, and incident response. Skilled in optimizing streaming platforms for throughput, latency, scale, and cost.
• Data Security and Governance - Strong knowledge of implementing data security at rest and in motion, encryption standards, access controls, and compliance — ideally within regulated financial environments (e.g., PCI-DSS, SOX, GDPR, data residency).
• Client Engagement and Delivery Leadership - Ability to bridge the gap between business needs and technical solutions, ensuring alignment on architecture, scalability, security, and outcomes. Excellent communication and stakeholder-management skills; able to operate independently in a fully remote, client-facing consulting context.
• Preferred / Nice-to-Have Direct experience in banking, payments, or financial services. Confluent certification (Developer or Administrator) and deep experience with Confluent Platform / Confluent Cloud. Experience with CDC tooling (Debezium), Infrastructure-as-Code (Terraform), and event-driven microservices. Proficiency in a JVM language (Java or Scala); comfort with Python a plus.

eye