We Proved Multi-Cluster Kafka Works on Kubernetes… Here's Everything We Learned

Why This Problem Matters

Apache Kafka has become the backbone of real-time data infrastructure, but its Kubernetes deployment model has a hidden assumption baked in: everything runs in one cluster.

When that cluster goes down — and at scale, it will — your entire Kafka deployment goes with it. MirrorMaker 2 gives you asynchronous replication to a second cluster, but it's not seamless. You get eventual consistency, not strong durability. You get failover with downtime, not transparent continuity.

The question I wanted to answer was: can a single logical Kafka cluster span multiple independent Kubernetes clusters, each with its own control plane, so that losing one cluster entirely doesn't bring Kafka down?

This is the stretch cluster pattern, borrowed from how databases like Cassandra distribute across data centers… applied to Kafka on Kubernetes. Nobody had done it with Strimzi (CNCF Incubating) before. We decided to find out if it was possible.

I presented this work at StrimziCon 2025 and Confluent Current 2025. A video walkthrough is also available on YouTube.

The Architecture

Why KRaft Changed Everything

This project became feasible specifically because of Apache Kafka's KRaft mode. ZooKeeper's operational model made cross-cluster deployments essentially impossible — you would have been fighting two distributed consensus systems across a high-latency link. KRaft's integrated Raft-based consensus protocol gave us a clean path.

The Three-Cluster Design

Central Cluster (Cluster A)
├── Strimzi Cluster Operator (manages all three clusters)
├── Kafka CR + KafkaNodePool CRs (the source of truth)
├── Brokers: IDs 0, 1, 2
└── KRaft Controllers: IDs 3, 4, 5

Remote Cluster B
├── Local Strimzi Operator (reconciles StrimziPodSets only)
├── Brokers: IDs 6, 7, 8
└── KRaft Controllers: IDs 9, 10, 11

Remote Cluster C
├── Local Strimzi Operator (reconciles StrimziPodSets only)
├── Brokers: IDs 12, 13, 14
└── KRaft Controllers: IDs 15, 16, 17

Central control plane model: Only one cluster runs the full Strimzi Cluster Operator with the Kafka and KafkaNodePool CRs. Remote clusters run a constrained Strimzi instance that only reconciles StrimziPodSets — it never touches the cluster-level configuration. This avoids split-brain in the operator layer.

The Plugin Architecture

Different environments have different networking constraints, so we built a plugin system for the cross-cluster connectivity layer. The plugin interface is intentionally minimal — each plugin implements address resolution and service lifecycle management. Adding a new networking backend is straightforward.

The Hard Problems (and How We Solved Them)

Problem 1: Cross-Cluster DNS

Kafka's advertised.listeners is where everything starts. Each broker advertises the address that clients and other brokers use to reach it. In a single-cluster deployment, this is a Kubernetes service DNS name. Across clusters, you need an address that resolves everywhere.

The Multi-Cluster Services API (KEP-1645) defines .clusterset.local as the DNS namespace for cross-cluster services. We implemented a naming scheme that generates stable cross-cluster DNS names:

<pod-name>.<cluster-id>.<service>.<namespace>.svc.clusterset.local

The Strimzi operator was modified to generate these names when stretch cluster mode is enabled, and to populate advertised.listeners with them instead of standard in-cluster DNS names.

The tricky part: Strimzi generates broker addresses during reconciliation, before the pods exist. The DNS names need to be deterministic and consistent across reconciliation cycles. We added a cluster-alias annotation to KafkaNodePool resources that provides the stable cluster identifier:

apiVersion: kafka.strimzi.io/v1
kind: KafkaNodePool
metadata:
  name: broker-pool-cluster-a
  labels:
    strimzi.io/cluster: my-stretch-cluster
  annotations:
    strimzi.io/stretch-cluster-alias: "cluster-a"
spec:
  replicas: 3
  roles:
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false

Problem 2: TLS Certificate SANs

Strimzi's internal certificate authority generates TLS certificates for all broker-to-broker and controller-to-controller communication. These certificates need Subject Alternative Names (SANs) that cover every address a broker might be reached at, including the cross-cluster DNS names.

The standard Strimzi certificate generation only knew about in-cluster addresses. We extended it to include .clusterset.local SANs for every broker in every remote cluster.

This is also directly relevant to open issue #11291 about removing OpenSSL from the certificate manager — the custom SAN generation logic I implemented gives practical insight into exactly what the Java Security framework replacement needs to handle.

Problem 3: KRaft Controller Quorum Across High-Latency Links

KRaft's Raft consensus protocol is sensitive to latency. In a single datacenter, controller-to-controller communication happens in sub-millisecond time. Across clusters, you're looking at 2–10ms depending on geography.

The controller.quorum.voters configuration needs all controller addresses specified statically at startup. We configured this with the full cross-cluster DNS names for all nine controllers:

controller.quorum.voters=\
  3@controller-3.cluster-a.controllers.kafka.svc.clusterset.local:9090,\
  4@controller-4.cluster-a.controllers.kafka.svc.clusterset.local:9090,\
  5@controller-5.cluster-a.controllers.kafka.svc.clusterset.local:9090,\
  9@controller-9.cluster-b.controllers.kafka.svc.clusterset.local:9090,\
  10@controller-10.cluster-b.controllers.kafka.svc.clusterset.local:9090,\
  11@controller-11.cluster-b.controllers.kafka.svc.clusterset.local:9090,\
  15@controller-15.cluster-c.controllers.kafka.svc.clusterset.local:9090,\
  16@controller-16.cluster-c.controllers.kafka.svc.clusterset.local:9090,\
  17@controller-17.cluster-c.controllers.kafka.svc.clusterset.local:9090

The tested configurations maintained quorum reliably at <10ms inter-cluster latency. Beyond ~50ms, leader election stabilisation time increases noticeably — worth knowing before planning a geographically distributed deployment.

Problem 4: Operator Resource Lifecycle Across Clusters

The central Strimzi operator needs to create, update, and delete resources in remote clusters without having administrative access to those clusters' Kafka resources. We implemented this using scoped kubeconfig secrets: the central operator has access only to the specific namespaces in remote clusters where Kafka resources live.

The operator tracks remote resources using a finalizer-based lifecycle:

# Annotation added to all remote resources
annotations:
  strimzi.io/stretch-cluster-alias: "cluster-b"
  strimzi.io/managed-by-central: "my-cluster"

When the central Kafka CR is deleted, the finalizer ensures remote StrimziPodSets are cleaned up before the central resources are removed.

What the Tests Showed

We ran three categories of failure tests. These are real results from the test environment — not simulations.

Broker Leader Failover

Setup: 18-partition topic, replication factor 9 (each partition replicated across all 9 brokers), producers running continuously at 1,000 msg/sec.

Controller Quorum Failover

Starting state: 9 controllers across 3 clusters, leader = Controller 5 (Cluster A).

Cross-cluster leader election worked. The ~1.2s election time when the quorum majority shifted to a remote cluster is within KRaft's expected operating range.

Full Cluster Failure

The scenario that matters most: all pods in Cluster A are terminated simultaneously.

Kafka remained operational. Producers and consumers connected to the bootstrap addresses of Clusters B and C continued without interruption. The 6 remaining brokers covered all partitions. The 6 remaining controllers maintained quorum and elected a new leader within 1.5 seconds.

This is the result that proves the core thesis: a Kafka cluster can survive the complete loss of one Kubernetes cluster, transparently, with no client reconfiguration.

Network Performance: Submariner vs Cilium

For latency-sensitive deployments, Cilium's eBPF approach measurably outperforms the userspace WireGuard path in Submariner. For most Kafka workloads the difference is not material, but it's worth knowing.

Cross-Community Engagement

This project required working simultaneously with four CNCF communities — something I'd encourage anyone building at the intersection of projects to do early and openly.

• Strimzi (CNCF Incubating): I submitted two formal proposals to the strimzi/proposals repository. The feedback from maintainers — particularly around operator complexity and testing surface — shaped the plugin architecture significantly. Even when a proposal doesn't get merged, the review process forces you to think through failure modes you'd otherwise miss.
• Submariner (CNCF Sandbox): The Submariner team's guidance on ServiceExport/ServiceImport patterns and ClusterSet DNS was essential. Working through the gap between the MCS API specification and Submariner's actual implementation taught me more about the state of multi-cluster networking than any documentation could.
• Cilium (CNCF Graduated): Cilium's ClusterMesh gives you the same MCS-compatible networking with an eBPF data path. Testing both gave me a side-by-side comparison that wouldn't be possible otherwise. The Cilium community's documentation on eBPF-accelerated cross-cluster routing is excellent.
• Fabric8 Kubernetes Client: Building multi-cluster client patterns in Java, managing multiple kubeconfig contexts, handling credential rotation, scoping API access per cluster — was not well documented. The patterns I developed are reusable for any Java operator that needs to manage resources across cluster boundaries.

Patterns That Apply Beyond Kafka

The architectural patterns from this project aren't Kafka-specific. If you're building or operating other stateful distributed systems on Kubernetes, these are directly applicable.

• Central control plane, distributed data plane: One operator owns the desired state. Remote clusters only reconcile the resources the central operator pushes to them. This prevents split-brain at the operator layer without requiring complex leader election between operator instances.
• Deterministic cross-cluster naming at reconciliation time: Any operator that needs to reference resources across clusters needs stable, pre-computable names. Don't generate names dynamically at runtime; the cluster annotation pattern gives you names you can compute before the resource exists.
• MCS API for service discovery: The .clusterset.local DNS namespace is emerging as the standard for cross-cluster service discovery. If you're building something that needs to work across clusters today, design your service discovery around this, even if you implement it with NodePort or LoadBalancer as a fallback.
• Scoped cross-cluster credentials: Don't give your central operator cluster-admin on remote clusters. Generate scoped kubeconfigs that grant access only to the specific namespace and resources the operator needs to manage. Use finalizers to ensure cleanup ordering.
• Pluggable networking backends: The cross-cluster networking technology changes faster than your application logic. Abstracting it behind a plugin interface means you can swap Submariner for Cilium (or for a future MCS implementation) without touching your core operator code.

What's Available

All code and documentation from this project are publicly available:

The State of Multi-Cluster Kafka

The Strimzi community's decision not to adopt this pattern into the main operator at this time is a reasonable one — the operational surface is real and the use case needs more production validation before it belongs in the core. That doesn't change what the prototype demonstrated.

Multi-cluster Kafka on Kubernetes is technically feasible today. The networking primitives (MCS API, Submariner, Cilium ClusterMesh) are mature enough. KRaft has removed the ZooKeeper constraint. The operator patterns are worked out.

What's needed now is production deployments that validate the operational model and build the community knowledge base around it. If you're running Kafka on Kubernetes at scale and thinking about cluster-level resilience, the code above is a working starting point.

If you're working on multi-cluster architectures — Kafka or otherwise — I'd genuinely like to hear from you. Find me on CNCF Slack in the #strimzi channel, or open an issue on any of the repositories above.

---

Acknowledgements

To the Strimzi maintainers who reviewed the proposals with genuine technical depth, especially the feedback that led to the plugin architecture. To the Submariner and Cilium communities for patient answers to questions that were sometimes very specific. And to everyone at StrimziCon 2025 who engaged with this work and pushed back on the parts that weren't fully thought through.