Kafka Engineer

Job Title: Kafka Engineer

Location: Pleasanton, CA (onsite)

Job Description:

Below is the JD, appreciate if you can share the profiles on priority.

Kafka Confluent Data engineer requirement.

Platform Design & Build

• Design and operationalize a multi-environment Confluent Cloud footprint (Prod/Non-prod) including RBAC, Service Accounts, API Keys, Private Networking (VPC peering/Private Link), IP allowlists, and encryption.
• Define topic architecture standards (naming, partitions, replication, retention, compaction), consumer group strategy, and exactly-once/at-least-once semantics.
• Implement Schema Registry and Stream Governance (compatibility modes, lifecycle, breaking change controls).
• Stand up connectivity via Confluent Cloud Connectors
• Establish Terraform/IaC modules for repeatable provisioning (environments, clusters, topics, ACLs, connectors, schemas).

Security, Compliance & Reliability

• Partner with Security to implement least-privilege, RBAC, secrets management, and compliance (PII, PCI/SOX/GxP as applicable).
• Define and enforce data governance (ownership, contracts, lineage, retention, DLP).
• Build SLOs/SLAs, error budgets, and incident response runbooks; coordinate with SRE for DR, multi-zone resilience, and capacity planning.

Observability & Operations

• Instrument metrics, logs, traces across Confluent Cloud and clients (lag, consumer health, throughput, latency, error rates, dead letter queues).
• Integrate platform telemetry with Datadog/PrometheGrafana/New Relic/Splunk; set alerts and automated remediation where feasible.
• Manage upgrades, quotas, cost optimization, and platform hygiene (ACL audits, orphaned topics/connectors cleanup).

Application Onboarding & Enablement

• Partner with app teams to design event contracts, idempotent producers, retry & DLQ patterns, and backpressure handling.
• Provide reference implementations and client libraries/configs (Java, .NET, Python, Node) with secure defaults and resiliency patterns.
• Run an onboarding desk : intake, discovery, architecture review, capacity estimates, provisioning via Terraform pipelines, and go-live checklists.
• Conduct training, documentation, and office hours; champion best practices for event-driven architectures.