Skip to content
Danube Messaging

Crash Recovery

This page explains what happens when a Danube broker crashes and how the system recovers without losing messages.

When Can Crashes Lose Messages?

Section titled “When Can Crashes Lose Messages?”

When a broker receives a message, it first writes to the local WAL (Write-Ahead Log) on disk. A background exporter then asynchronously uploads sealed WAL segments to Cloud Storage (S3, GCS, or Azure Blob).

The crash behavior depends on the underlying storage infrastructure:

  • Persistent Storage (Standard PVCs or Bare-Metal): If a broker process crashes, restarts, or is rescheduled on the same persistent volume, no messages are lost. The local WAL is intact, the broker replays the log tail upon startup, and normal operation resumes with zero data loss.
  • Ephemeral Storage (Spot Instances, Diskless Pods): If a broker runs on disposable compute where node termination permanently destroys the local disk, any messages written to the local WAL that were not yet sealed and uploaded to Cloud Storage would vanish with the node.

This is the specific problem Valkey solves. By buffering in-flight messages in Valkey in real-time, the cluster maintains a transient off-node copy that survives complete node and disk destruction on ephemeral infrastructure.

Recovery With Valkey

Section titled “Recovery With Valkey”

When a broker starts and takes ownership of a topic, it checks Valkey for any buffered messages that aren’t yet covered by Cloud Storage. If found, it replays them into a fresh WAL.

  1. Broker starts and claims topic ownership from the metadata store

  2. Check Valkey for the topic’s stream, query for any messages with offsets beyond what Cloud Storage already covers

  3. Download and replay, if Valkey has buffered messages, download them via XRANGE and replay them into a fresh local WAL

  4. Resume normal operation, the WAL now contains all recovered messages. New writes continue from the correct offset. Consumers reconnect and see a continuous stream.

From the consumer’s perspective, nothing happened. The offset sequence is continuous, no messages are missing, and reads work normally.

Recovery Without Valkey

Section titled “Recovery Without Valkey”

Without the write buffer, the system depends entirely on the local WAL and Cloud Storage:

  • Messages already exported to Cloud Storage: safe, always recoverable
  • Messages in the WAL but not yet exported: lost if the disk fails or the broker crashes

The window of exposure depends on:

  • WAL rotation frequency (default: every 512 MB)
  • Cloud Storage export cadence (continuous background task)

In practice, this window is typically a few minutes of recent messages.

Recovery During Topic Moves

Section titled “Recovery During Topic Moves”

When a topic moves between brokers (due to rebalancing or maintenance), the system follows a coordinated handoff:

  1. Old broker seals the topic: stops accepting writes, records the last committed offset, and exports all remaining WAL data to Cloud Storage

  2. New broker loads the topic: reads the sealed state from metadata, creates a fresh WAL starting at last_committed_offset + 1

  3. Valkey is not involved: because producers were stopped during sealing and all data was exported to Cloud Storage, there is no gap for Valkey to fill. Valkey’s crash recovery role applies only to unplanned broker failures.

  4. Consumers reconnect: they resume from their subscription cursor. The tiered reader serves historical data from Cloud Storage, then transitions to the new broker’s WAL for live data.

For more details on topic moves, see Reliable Topic Move.

Guarantees by Configuration

Section titled “Guarantees by Configuration”

The table below contrasts durability guarantees between persistent volume deployments and disposable ephemeral compute:

ConfigurationBroker Restart (Persistent Disk / PVC)Node Destroyed (Ephemeral Disk Vanished)Valkey DownOverall Guarantee
WAL onlyZero message loss (WAL survives on disk)Unexported messages lostN/ALocal durability
WAL + Cloud StorageZero message lossUnexported window lost; all sealed safeN/ARecommended for PVCs (Zero external deps)
WAL + ValkeyZero message lossZero message loss (Replayed from Valkey)In-flight lostRPO=0 on ephemeral nodes (no cloud)
WAL + Valkey + Cloud StorageZero message lossZero message loss (Replayed from Valkey)Cloud storage safeFull enterprise durability on spot compute

What Each Tier Protects Against

Section titled “What Each Tier Protects Against”
RiskWALValkeyCloud Storage
Broker restart (same volume)SurvivesSurvivesSurvives
Broker lost (new instance, new volume)LostSurvivesSurvives
Valkey primary crashSurvivesLost (unless replicated)Survives
Cloud Storage region outageSurvivesSurvivesUnavailable