# Durability {#durability}


What an answer from the engine promises, what survives which failure, and
where the limits are.


## What an acceptance means {#what-an-acceptance-means}


The engine answers a submission, over SMTP or the SES API, only once the
message, its body and a delivery for each recipient are committed to its
store with a synced write. From that answer on, the message survives a
crash of the process, a loss of power, and a restart: it is delivered,
bounced or expired, and never silently dropped. Before the answer, nothing
was promised, and nothing is left behind.

The store is one SQLite database in write-ahead logging mode, with every
commit synced, and one writer that groups concurrent commits into one
sync. Deliveries, attempts, outcomes, the event journal, suppression, policy
and the dispatch ledger are all in it, so each change is one transaction:
no state lives only in memory that a restart could lose, other than what
can be rebuilt from the store.


## What each failure leaves {#what-each-failure-leaves}


| Failure | What follows |
|---|---|
| Before an acceptance commits | No success was returned; nothing is kept |
| After it commits, with the answer lost | The message exists; a client that retries sends a second one |
| The process dies before or during an attempt | The attempt is recovered as uncertain and tried again |
| A receiver accepts, and the engine dies before recording it | The message may have been delivered; trying again can deliver it twice, and the attempt is counted as uncertain |
| An event subscriber is down | Sending is unaffected; events are retried, parked, and replayed on request, for as long as the journal keeps them (below) |
| A subscriber handles an event and its answer is lost | The same event, with the same ID, is posted again |
| The store reports corruption | The affected work stops and the evidence is kept; nothing is skipped or repaired into an apparent success |
| The disk fills | The store stops, answering nothing it could not keep; at most the commit that failed is in doubt, and the engine starts sound once there is room |

The one unavoidable window is SMTP's own: the receiver's final reply hands
it the message, and a reply lost after that leaves the outcome unknown. No
local transaction makes that exchange exactly once, so the engine delivers
at least once, and counts each uncertain attempt
(`nitromta_attempts_total{result="uncertain"}`) so the window is visible.


## How it is tested {#how-it-is-tested}


- **Power loss.** A simulated disk and a pipeline of acceptance, dispatch,
  receiver and outcome fail the power at every I/O operation, then recover,
  and check every acknowledged commit after each recovery. A receiver sees a
  delivery twice only when an uncertain attempt was counted. The same runs
  stop the engine gracefully, and fill the disk, at random points.
- **Process loss.** The release binary, sending 500 messages a second, is
  killed and started again: in each run, about 4,960 recipients had been
  acknowledged and 4,840 were still queued at the kill, none was lost and
  none delivered twice, and the engine was accepting again 0.4 seconds
  after it started.
- **Host loss.** A message accepted and held on one host, its store on a
  volume of its own; the host powered off; the volume moved to a fresh host
  with the package installed and the configuration and keys restored from
  their copy; the engine started there holding the delivery, and sent it.
- **Corruption.** A store left open by a crash is checked as the engine
  starts, and a damaged one refused.

The power-loss and simulation tests run in `cargo xtask check`; the process
loss runs in the benchmark harness (`cargo harness kill`).


## Backup and restore {#backup-and-restore}


`nitromta backup --config <file> --to <file>` copies the store while the
engine runs, a few pages at a time between its own commits, or while it is
stopped. `nitromta restore` puts a copy in place of a stopped engine's
store, fenced: every delivery the copy holds that was not finished when it
was taken is held, since the original may have sent it since, until the
operator decides, with `nitromta fence --config <file> release` to send
them or `nitromta fence --config <file> discard` to cancel each as it falls
due.

The secrets in a backup, as in the store, are sealed under the
installation's keys, which are never in it, and so are message bodies while
encryption is on: keep a copy of the keys and the configuration apart from
both the host and its backups.


## The limits {#the-limits}


- **One host.** The store is on one host's disk. Losing the disk loses
  what was accepted since the last backup; there is no replication and no
  failover. A volume that outlives its host, as cloud block storage does,
  can be attached to another host and started there.
- **At least once.** See SMTP's window above.
- **No idempotency key.** A client that retries a submission whose answer
  was lost sends a second message, as with SES.
- **Events are kept for the feedback window.** The journal, and the
  batches parked from it, are kept for `[store] feedback_window_seconds`
  (seven days by default). A
  subscriber down for longer misses what was trimmed meanwhile, counted in
  its cursor's skipped positions, and a parked batch past the window can
  no longer be replayed.
- **The disk reserve is advisory.** New mail is refused once free space
  falls below `[store] reserve_mib`, but mail already accepted, outcomes,
  the log and checkpoints can still fill the disk; then the store stops,
  answering nothing it could not keep, until it is restarted with space
  free.