Hot: Local WAL
Fast, local disk write-ahead log. Every reliable message is written here first. The source of truth for all recent reads. Retained for hours to days based on time/size retention policies.
Durability & Storage Overview
Danube is designed to be a lightweight streaming platform with enterprise-grade durability. Brokers hold local state (the WAL) but do not replicate message data across nodes, durability beyond the local disk is delegated to purpose-built external systems. This section explains how Danube persists messages, the architectural decisions behind the design, and what it means for operators.
Danube supports two dispatch strategies: non-reliable (pub/sub, fire-and-forget) and reliable (streaming with at-least-once delivery). The durability features described here apply only to reliable topics.
How Danube Approaches Persistence
Section titled “How Danube Approaches Persistence”Reliable topics use a Write-Ahead Log (WAL) on the broker’s local disk as the foundation of all message persistence. Every message is appended to the WAL and assigned a monotonic offset before it is dispatched to consumers. The WAL provides fast, sequential writes, durable storage that survives broker restarts, and the offset-based replay that powers subscription cursors.
On top of the WAL, Danube offers two optional layers that extend durability beyond the local broker:
-
Valkey/Redis (Optional): transient write buffer for ephemeral compute. When running on disposable or spot nodes without persistent disks, messages can be double-written to Valkey Redis Streams. If an ephemeral node vanishes before S3 export, a replacement broker replays buffered messages from Valkey into a fresh WAL, achieving RPO ≈ 0. For deployments with persistent volumes (PVCs or local NVMe), the local WAL guarantees durability across restarts directly without needing Valkey.
-
Cloud Object Storage (S3/GCS/Azure Blob): long-term archival. A background exporter uploads sealed WAL segments as immutable objects for indefinite retention and full historical replay.
This layered approach keeps the broker lightweight, it does not need to replicate data via Raft or maintain expensive NVMe SSDs, while achieving enterprise-grade durability through purpose-built external systems. Scaling up and down becomes simpler because historical data lives in durable storage, not on the broker’s local disk.
Danube vs. Traditional Streaming Platforms
Section titled “Danube vs. Traditional Streaming Platforms”Most streaming platforms achieve durability by making their brokers stateful. Each broker stores replicated message data on local disk, typically using an embedded consensus protocol like Raft. This works, but it has consequences:
- Brokers need persistent storage (often fast NVMe SSDs) on every node
- Scaling down requires waiting for data to migrate off the node
- Scaling up requires waiting for data to rebalance onto the new node
- Cluster operations become storage operations
Danube takes a different approach. Brokers remain lightweight routers with a local WAL. The “hard problem” of cross-node durability is delegated to systems built specifically for it:
| Traditional Streaming Platforms | Danube | |
|---|---|---|
| Brokers | Stateful, each broker stores replicated data via Raft | Lightweight, brokers write to a local WAL and delegate cross-node durability |
| Durability | Embedded in every broker (local SSD + Raft consensus) | Local WAL + optional Valkey write buffer + optional Cloud Storage |
| Storage | Persistent disks required on every node | Ephemeral disks sufficient; durable history lives in Cloud Storage |
| Scaling up | Wait for data to rebalance onto new nodes | Start a new broker, it reads historical data from durable storage |
| Scaling down | Wait for data to migrate off the node | Stop the broker; durable segments in Cloud Storage are already available to other brokers |
This is architecturally closer to Apache Pulsar (stateless brokers + stateful BookKeeper), but dramatically simpler. Where Pulsar requires a ZooKeeper cluster + a BookKeeper cluster + Pulsar brokers (9+ processes minimum), Danube requires a Valkey instance + Danube brokers. And Valkey is something most teams already operate for caching.
The Three Storage Tiers
Section titled “The Three Storage Tiers”Danube organizes message storage into three independent tiers, each with its own backing store, retention policy, and performance profile:
Warm: Valkey
Optional Redis Streams write buffer. Messages are double-written to Valkey for crash recovery. Provides fast reads of recent data when the local WAL is lost. Retained for the last N closed segments.
Cold: Cloud Storage
Optional S3/GCS/Azure Blob archival. Sealed WAL segments are exported as immutable objects. The authoritative source for historical reads and broker-to-broker topic moves. Retained indefinitely.
Each tier is independent. You can run any combination:
| Configuration | Tiers | Use Case |
|---|---|---|
| WAL only | Hot | Development, testing, edge nodes with persistent disk |
| WAL + Cloud Storage | Hot + Cold | Production on Persistent Volumes (PVCs / NVMe) + Cloud archival (zero external deps) |
| WAL + Valkey | Hot + Warm | Crash recovery buffer on ephemeral nodes without cloud archival |
| WAL + Valkey + Cloud Storage | Hot + Warm + Cold | Production on Ephemeral / Spot compute with Cloud archival |
Storage Modes
Section titled “Storage Modes”The storage.mode setting controls where sealed WAL segments are exported for durable, cross-broker availability:
| Mode | Durable Storage | Use Case |
|---|---|---|
local | Local filesystem only | Single-node, development, edge |
shared_fs | Shared filesystem (NFS, EFS) | On-prem multi-broker clusters |
object_store | Cloud object storage (S3, GCS, Azure Blob) | Cloud-native multi-broker deployments |
In all modes, the local WAL handles recent writes and real-time reads. In shared_fs and object_store modes, a background exporter continuously publishes sealed segments to durable storage for historical replay and cross-broker availability.
When a topic moves between brokers (e.g., during rebalancing), the new broker reads historical data from the durable storage backend. In local mode, only the offset sequence is preserved, there is no shared durable backend. For the full move protocol, see Reliable Topic Move.
See Broker Configuration for the full YAML reference.
What Users Get
Section titled “What Users Get”Reliable topics without Valkey or Cloud Storage: The local WAL provides durability across broker restarts. Messages survive as long as the local disk is intact. No crash recovery if the disk is lost.
Add Valkey:
- Zero-message-loss crash recovery (RPO ≈ 0, quorum-confirmed writes)
- Fast failover, a replacement broker downloads buffered messages from Valkey and replays them into a fresh WAL
- Warm cache for historical reads when local WAL files have been pruned by retention
Add Cloud Storage:
- Indefinite message retention
- Full historical replay from any offset
- Cross-broker data availability for topic moves and rebalancing
- Complete data durability even if both the broker and Valkey lose data simultaneously
Next Steps
Section titled “Next Steps” Tiered Storage How the read and write paths work across Hot, Warm, and Cold tiers with invariants and examples.
Configuration Guide Practical YAML examples for enabling each tier.
Crash Recovery What happens when a broker crashes and how Valkey ensures zero message loss.