Skip to content

Run a failover

Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: none, retired with Core.

Promote a replica to primary when the primary is offline. The primary is the only host that publishes; while it is gone reads continue from every replica on its own hostname, and writes wait. Fleet owns the role model and how automatic promotion selects and arms; this page is the two procedures you run.

Symptoms

  • The health sweep posts Host <town>: Brak odpowiedzi, and the apex hostname returns a Cloudflare 530 (a tunnel with no connector).
  • Sessions cannot be closed. Replicas still answer, read-only.

Decide first: wait or promote

A sleeping laptop is the common case, and it usually returns within minutes. Promote only when the primary will be gone longer than the work can wait. Promotion has a real cost: gitignored private state (tokens, capability grants, the PU ledger cache) does not travel with the role, so every narrator re-authenticates and every grant is re-issued on the new primary — only API-key automation (Permissions) rides the committed store and keeps working unchanged.

Manual promotion: the fast path

When a human is present this beats waiting out the automatic net, and it is the only path for a fleet with no replica on takeover: auto.

  1. Open a merge request against the fleet repo swapping the two role: values in fleet/hosts.yaml: the old primary becomes replica, the chosen replica becomes primary. Exactly one primary may remain; validation refuses anything else.
  2. Check fleet:plan on the merge request. It must show the apex moving and nothing else.
  3. Merge. fleet:provision re-points the apex CNAME at the new primary's hostname, re-renders both descriptors, and reconciles the lore deploy keys so the new primary's key is the one that can push.
  4. Both devices converge within five minutes. The new primary starts publishing; the old one drops to converge-only on its next tick if it is alive.
  5. Confirm: the apex answers, and the next sweep reports the new primary reachable.

The automatic net

When nobody is awake, the health worker arms a takeover on its own, opens the same role-flip merge request titled Zmiana hosta APEX: <primary> → <town>, sets it to merge on a green pipeline, and posts the link to the channel. Fleet owns which replica it picks and how long it waits first. Each takeover advances an epoch counter, tracked internally rather than in the message text; Manage notifications lists the takeover lines. End to end the automatic path runs about 40–55 minutes, so the manual flip above is faster whenever someone is at a keyboard. Only a replica carrying takeover: auto is eligible.

The net rides on the sweep. While the worker stands down behind the standby gate it runs no sweep and arms no takeover, and the monitor host owns the watch instead (Stand the health worker down is that cutover).

Before proposing, every proposer runs the pre-flight: it lists standing takeover/* merge requests, adopts an open one, and honours one merged inside the cooldown — so overlapping proposers converge on one MR instead of racing a two-primary roster into main (Fleet explains why this listing is the only guard).

Requesting the flip through the mesh. An operator (or the monitor) can also ask a daemon to propose the flip over the API — same ranking, same pre-flight, same cooldown, same auto-merge window, with the requesting host's own peer observations embedded in the MR description as the justification. Unforced requests demand that corroboration; force waives only the corroboration (for split-brain repair) and never the pre-flight or the cooldown. The wire contract lives in the API reference.

After any flip — automatic, manual, or mesh-requested — run the dispatch-credential step: Rotate secrets → The daemon dispatch token, "after a takeover". Until it runs, Discord closes fall back to CI on every attempt (functional, but the fast path stays dark).

Disable it mid-incident

When an outage is understood and you are handling it by hand, turn the net off so it does not fire underneath you. It is a one-line merge request in fleet.settings.yml:

auto_failover:
  enabled: false

It takes effect on the next provision and sweep; the worker then alerts and proposes nothing. Turn it back on the same way.

If the old primary comes back

Nothing breaks. It converges, reads role: replica, and boots read-only; its lore deploy key lost its write bit on the provision run that demoted it. A session it staged locally is still on origin as staging/<sid> and can be adopted on the new primary.

For up to one converge tick (≤5 minutes) after provision re-renders it, the recovering host still believes it is primary and two hosts hold publish rights. The landing gate makes a stale publish either fast-forward cleanly or lose the compare-and-swap, so confirm the window closed:

  • git log --oneline on main around the flip: a clean sequence, not a pair of racing commits for one session.
  • The closed and intel echo union (nerthus.closed.*.md, nerthus.intel.*.md): one entry per session header, no duplicate across host files.
  • The demoted host's key: can_push is false again on the lore project, so it cannot publish even if it briefly tried.

Do not

Do not move the apex by editing DNS by hand. The record is ledgered in state/fleet.lock.json, and a hand-made record fails the next provision's ownership read-back.

See also