KIP-1150, driven by the team at Aiven, proposes Diskless Kafka — an architectural shift that replaces local broker disk storage with pluggable cloud object storage (S3, GCS). Brokers become stateless, scaling becomes elastic, and entire classes of disk-related operational failures disappear. If adopted, Kafka becomes one of the first open-source platforms to support both traditional disk-based and fully cloud-native diskless topics.
This post reflects my personal interpretation and analysis of KIP-1150. It's not meant to be definitive or exhaustive — just my way of making sense of an exciting proposal. I'd love to hear your thoughts and continue the conversation! 😊
The Cloud Storage Imperative
Modern cloud infrastructure operates with fundamentally different assumptions than the on-premises environments Kafka was originally built for. In the cloud, local disks are often ephemeral, provisioning capacity ahead of time is inefficient, and managing replication across brokers increases complexity and cost.
Diskless Kafka addresses these challenges by eliminating the broker's dependency on local storage. Instead of writing data to disk and replicating it across brokers, diskless topics persist records directly to a pluggable object store — such as Amazon S3 or Google Cloud Storage — at the moment of ingestion. This transforms Kafka into a system that naturally aligns with cloud-native principles: elastic, fault-tolerant, and pay-as-you-go.
The implications are significant. It removes the need for careful disk sizing, dramatically reduces IOPS-related infrastructure costs, and eliminates entire classes of operational failures tied to local storage. Brokers become stateless in the traditional Kafka sense, allowing them to be started or terminated rapidly without data movement or risk.
With object storage taking on the role of the durable commit log, Kafka can scale more like stateless services — without compromising on durability or availability guarantees.
How Diskless Kafka Works Under the Hood
At its core, Diskless Kafka redefines how brokers handle data persistence. When producers send messages, brokers now keep the data in memory rather than writing to local disks. This in-memory buffer gets asynchronously flushed to cloud storage, with writes only acknowledged after successful persistence.
Consumers continue reading seamlessly — recent data comes from broker memory for low-latency access, while historical data is fetched directly from cloud storage when needed.
in-memory buffer
S3 / GCS
This architectural shift brings immediate operational benefits. Broker restarts that previously took minutes (while reloading disk-based logs) now happen in seconds. Cluster scaling becomes dramatically simpler since new brokers don't need to replicate partition data locally. And perhaps most importantly, it eliminates the operational headaches of managing disk failures, capacity planning, and performance tuning of local storage.
Because any broker can accept writes for the same partition, clients no longer need to route writes to a specific leader. This simplifies client logic, reduces cross-zone traffic, and eliminates downtime during leader failover — a radical break from Kafka's traditional leader-follower architecture.
Tiered Storage: The Foundation for Diskless Architecture
Diskless Kafka didn't emerge in a vacuum — it builds upon Kafka's Tiered Storage (introduced in KIP-405), which first separated compute from storage. Tiered Storage allowed older log segments to move to cheaper object storage while keeping recent data local. Diskless Kafka takes this further by removing local storage entirely.
The evolution makes perfect sense. Tiered Storage proved remote storage could work reliably for Kafka. Diskless Kafka simply completes this transition by making remote storage the single source of truth. This progression mirrors how cloud-native applications have evolved — first hybrid architectures, then full commitment to cloud paradigms.
Why This Matters for Real-World Deployments
The implications for production systems are significant. Diskless Kafka replaces costly local SSDs with pay-as-you-go object storage, which can dramatically reduce infrastructure costs. It also simplifies operations by eliminating entire classes of disk-related failures, replication logic, and recovery complexities.
For high-throughput, low-priority workloads or archival pipelines, diskless topics offer a way to scale Kafka more efficiently and cost-effectively — without over-provisioning storage.
Classic Topics vs Diskless Topics: Choosing the Right Tool
Diskless topics won't replace classic Kafka topics — they'll complement them. The right choice depends on your workload's requirements.
For ultra-low-latency, high-throughput use cases — like financial tick data or ML feature pipelines — classic topics with local disks may still reign. But for log analytics, audit trails, or compliance events, diskless topics offer massive cost savings and operational simplicity.
KIP-1150 is still a proposal, and the details will evolve as the community engages with it. But the direction is clear: Kafka is moving towards a future where storage is a cloud service, not a local concern. And that future looks compelling.
If you're following this proposal or have thoughts on the trade-offs, I'd love to hear from you.