Hot: Local WAL
Every message is appended to a local Write-Ahead Log (WAL) on the broker’s disk and assigned a monotonic offset. This is the fast path for all writes and real-time reads.
Persistence & Storage
Reliable topics persist messages so consumers can replay history and topics can move between brokers without losing data. Non-reliable topics bypass storage entirely.
Storage Architecture
Section titled “Storage Architecture”Danube’s persistence is built on three independent storage tiers:
Warm: Valkey
Optional. Messages are double-written to Valkey Redis Streams for crash recovery. If the broker is lost, the new broker replays buffered messages from Valkey into a fresh WAL.
Cold: Cloud Storage
Optional. A background exporter uploads sealed WAL segments to cloud object storage (S3, GCS, Azure Blob) as immutable segment objects for long-term archival and historical replay.
Each tier is independent and opt-in. You can run any combination depending on your durability and retention requirements.
Storage Modes
Section titled “Storage Modes”The storage.mode setting controls where sealed WAL segments are exported:
| Mode | Durable Storage | Use Case |
|---|---|---|
local | Local filesystem only | Single-node, development, edge |
shared_fs | Shared filesystem (NFS, EFS) | On-prem multi-broker clusters |
object_store | Cloud object storage (S3, GCS, Azure Blob) | Cloud-native multi-broker deployments |
In all modes, the local WAL handles recent writes. In shared_fs and object_store modes, a background exporter continuously publishes sealed segments to durable storage for historical replay and cross-broker availability.
Write Path
Section titled “Write Path”- The broker appends the message to the local WAL and assigns the next topic offset
- If Valkey is configured, the message is simultaneously written to a Redis Stream with replica confirmation
- The message is published into the in-memory cache and live reader stream
- Asynchronously, sealed WAL files are exported to cloud storage (in
shared_fsandobject_storemodes)
Read Path
Section titled “Read Path”When a consumer reads from a topic, the broker serves data from the tier closest to the source:
- Live tail consumers read directly from the WAL’s in-memory broadcast stream with zero overhead
- Historical consumers receive a stitched stream: Cold (cloud storage) → Warm (Valkey) → Hot (WAL), with each tier covering a non-overlapping offset range
The local WAL always takes precedence for data it still holds. Valkey and cloud storage are only consulted for offsets that have been rotated out of the WAL.
Topic Moves
Section titled “Topic Moves”When a topic moves between brokers (e.g., during rebalancing), offset continuity is preserved:
- The old broker seals the topic and records the last committed offset
- The new broker resumes from
last_committed_offset + 1
In shared_fs and object_store modes, historical reads on the new broker are served from durable segments. In local mode, only the offset sequence is preserved, there is no shared durable backend.
For the full protocol, see Reliable Topic Move.
Learn More
Section titled “Learn More” Durability & Storage Overview Why Danube uses stateless brokers with external durability, and how it compares to other platforms.
Tiered Storage How the read and write paths work across Hot, Warm, and Cold tiers with invariants and examples.
Configuration Guide Practical YAML examples for storage modes, Valkey write buffer, and cloud backends.
Crash Recovery What happens when a broker is lost and how each tier protects against data loss.