Restore the estate¶
Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: not yet written.
When gitlab.com does not have the data.
Every other recovery runbook in this tree works because it does. Bootstrap a fleet host clones from a working remote, Recover a diverged host re-clones from scratch, Offboard a host removes one. This page is for the other case.
Read this section before anything else. Some of it is not coming back.
These are recorded as accepted losses in
Nerthus.Core.Infra.Common/schema/recovery-objectives.yml. Do not spend the first hour of an
incident looking for them.
runtime/apikeys— the daemon's minted API keys. One copy, in a Docker volume. Re-mint them; they are cheap. Nothing restores them.staging/edits— unlanded contributor drafts. Deliberately per-host and deliberately not replicated (H97.5). A host loss destroys them and their author is not told by anything.log/audit.jsonl— the estate's only account of who did what. It exists on one host. Whether this is an accepted loss isundecided: the row has not been ratified.- Any image pinned more than about a day ago. See The image below.
- The tunnel tokens in
secrets/*.tunnel.age. They name a tunnel id that a recreated tunnel does not reuse. Recovery is afleet:provisionrun, not a restore. See The edge below.
The replicas hold none of the above. "We have three hosts" is a true statement about the corpus and a false one about these records.
The order, and why¶
A reader at 3 a.m. does not have time to derive the dependency graph.
- Credentials. Nothing else can start.
- Repositories. From the copy — see the warning under that step.
- The image. At the pinned digest.
- The host. Bootstrap it.
- The records. Or accept them as lost, per the box above.
- The edge. Tunnel and DNS.
0 · The accounts that cannot be re-minted¶
Everything else in this estate is re-mintable by somebody who can log in. These four are where the recovery is a support ticket and a human, and they are the long pole. Start them first, in parallel with everything below, because they are the only steps with a queue in front of them.
| account | why it is terminal |
|---|---|
| GitLab (gitlab.com) | every repository, all CI variables, deploy keys, and the container registry |
Cloudflare (d78ae8ef…) |
every tunnel, every DNS record, the Workers, the KV namespace |
the registrar for nerthus.pl |
a lapsed domain can be bought by somebody else |
| the Discord developer portal | the application id cannot be re-created; every guild binding is on it |
Nerthus.Core.Infra.Common/sejf/recovery-map.yml carries this list with the reasoning, and
recovery:map republishes it on a schedule. It says where the cold kit is and what is in
it — never what is inside the envelope.
1 · Credentials¶
Open the vault from a machine that is not the operator's laptop. SOPS_AGE_KEY is a file-type
protected CI variable on nerthus.core.infra.common, so a project Owner can decrypt the tree in a
job.
Settled 2026-08-27: SOPS_AGE_KEY holds the operator identity.
The sejf:identity job on nerthus.core.infra.common's default branch answers this every run,
by result rather than by inspection: it derives the public half of whatever CI holds and
matches it against the two recipients in .sops.yaml. It also decrypts a real file as a
positive control, and refuses with a freshly generated wrong identity as a bounded negative
control — 0 s, exit 128, a refusal and not the timeout a wrong credential can produce here.
The two halves survive different failures, and that is the useful part. Lose the laptop and the operator half is still in CI. Lose GitLab and the break-glass half is still on the offline media. What it costs: the operator's own daily identity is duplicated into a CI variable, so any Owner on that project can read the whole vault.
Run the job to re-confirm before trusting this page — the variable can be replaced without anything here changing.
If this step needs a value that is only on the operator's laptop, stop: the rest of the walk is documenting the size of the hole rather than closing it.
2 · Repositories¶
From the copy, not from gitlab.com.
Block egress to gitlab.com before you start, and check the block.
This is the single most important line on the page. Every restore tool in existence will try
the original remote, and a restore that quietly fell back to origin looks perfect and proves
nothing. Verify the block with a request that must fail:
docker network create --internal restore-net
docker run --rm --network restore-net --entrypoint sh alpine/git \
-c 'git clone --depth 1 https://gitlab.com/margonem/nerthus/it/nerthus.core.git /tmp/x'
# must fail: Could not resolve host: gitlab.com
Measured 2026-08-26 on eder: the failing control took 5.6 s (DNS timeout) and the positive control on the default network succeeded in 1.0 s. Run both. A block that fails because the network is down proves nothing either.
--internal blocks the container's traffic. It does not block an image pull: the
daemon pulls on the host. Pull your tooling image before you create the network, or the pull
will succeed and you will believe egress is open when it is not.
Then clone each repository out of the mirror:
docker run --rm --network restore-net --entrypoint sh \
-v /srv/nerthus-rehearsal/mirror:/mirror:ro -v /srv/restored:/restored alpine/git \
-c 'cd /restored; for g in /mirror/*.git; do git clone "$g" "$(basename "$g" .git)"; done'
Measured: 688 ms for four repositories (1 056 + 383 + 520 + 33 commits), with egress blocked.
Confirm each clone's origin points at the mirror and not at gitlab.com.
rev-list --count --all will be SMALLER on the restored clone. That is not data loss.
Measured: 1 017 against the mirror's 1 056, 332 against 383, 426 against 520. A --mirror
holds refs/merge-requests/* and other server-side refs that an ordinary clone does not
localise. rev-parse HEAD is the comparison that matters and it was identical for every
repository. Compare the ref map, not the count.
3 · The image¶
Nerthus.Core.Infra.Dev/versions.lock names it.
The bytes may already be gone, and this is the most likely step to fail outright.
Measured 2026-08-26: of the 86 digests ever written to versions.lock, 7 resolve. Two are
recent and five survive only because a feature branch left them a tag that was never reused.
GitLab.com collects untagged manifests on a roughly 24-hour cycle this estate did not configure
and cannot tune, and the project's own cleanup policy is disabled and has never run —
reading a retention off it gives a 90-day answer that is wrong by two orders of magnitude.
So: the fleet can break itself in 15 minutes and un-break itself for about 28 hours.
If image_previous: is present, fleet:rollback is the way back and it probes the registry before
it writes anything. If it is absent, this tree predates that field and there is no recorded way
back — the digests are in git log -p -- versions.lock and whether their bytes survive is a
registry question, not a git one.
Recovery beyond the window is a rebuild from the source commit. Nobody has ever rebuilt a three-week-old commit on today's CI, so that step has no measured duration.
4 · The host¶
Commission a rented host and Bootstrap a fleet host are the existing procedures.
The fleet repository is an input, and production does not exist yet
Bootstrap a fleet host used to clone https://gitlab.com/margonem/nerthus/narratorzy/nerthus.core.infra.prod.git
unconditionally - a project nobody has created - so its first command could not run. It now
takes the fleet repository as an explicit input and shows both rows. Project 98 owns what
production is; for a dev-fleet restore, use the dev row.
fleet:provision runs only on the self-hosted runners, and both are on eder. If eder is among
the dark hosts, CI cannot bring up the host CI needs, and the bootstrap above is the manual path.
5 · The records¶
Restore, or accept as lost per the box at the top. What a restored host must show:
| surface | the comparison |
|---|---|
| a repository | rev-parse HEAD equal to the source; ref map equal; then git diff between trees empty |
| the image | /health's imageDigest equal to the pinned digest |
| the daemon records | counts equal per path, then one record read back field by field |
| the corpus, served | a route that reads the corpus answers what the source host answers |
A restored host that git says is identical can still serve nothing.
versions.lock carries schema_version, and a host whose on-disk format outruns the pin goes
read-only rather than writing against a format it does not understand. A restore that
produces a read-only host is a partial restore. Record it as one, with the schema value on
both sides.
6 · The edge¶
The tunnel tokens in secrets/ cannot restore a deleted tunnel. Measured, not argued.
Settled 2026-08-26 on a throwaway tunnel. A cloudflared connector token decodes to three
fields — a (account tag), s (the secret) and t, the tunnel id. Create a tunnel with a
secret you choose, delete it, then recreate it with the same name and the same secret: you
get a new id. The token names an id that no longer exists.
So Nerthus.Core.Infra.Dev/secrets/*.tunnel.age is a copy of something that stops being true
the moment its tunnel is gone. Do not spend incident time trying to restore from it.
Recovery is re-provisioning, not restoring bytes. fleet:provision mints a token per host
and encrypts it to that host's age key — the estate can already do this. The durable half is the
ingress rules in the repository, which are what tell the new tunnel what to serve.
No measured duration: nobody has timed a provision against an account whose tunnels are gone.
What has been walked, and what has not¶
The authority of a runbook is that somebody did it. This page is written from 97/06's transcript
on 2026-08-26 and is explicit about the difference.
| step | walked? |
|---|---|
| 2 · repositories from a copy, egress blocked, both controls run | yes — 688 ms, four repositories, on eder |
| 3 · the pinned digest resolves | yes, and it was free: eder already held the layers |
| 3 · the previous digest | no — image_previous: did not exist on the tree yet |
| 0/1 · credentials, the vault opened off the laptop | yes, 2026-08-27 — sejf:identity opens it in CI and names which identity it used |
| 4 · host bootstrap, DNS, the tunnel | no — free on eder, which already has all of them |
| 5 · daemon records restored | no — no rehearsal daemon; the live replica was not touched |
| the ungraceful kill with an open staged edit | no — see below |
| 6 · the tunnel rebuild | settled 2026-08-26 — measured on a throwaway tunnel; the answer is that the stored token cannot restore one |
The cheap pass on eder gets three things wrong, honestly. eder already has the image layers, the runner, docker and network reachability, so steps 3, 4 and 6 were free rather than exercised. Those are precisely the steps a rented-box pass exists to test, and they are the ones that will be unfamiliar at 3 a.m.
The ungraceful kill was not performed. It needs a rehearsal daemon, which needs the credential
path in step 1, which did not run. Killing a live replica to produce the test would have measured
the fleet rather than the restore. All three live hosts hold 0 stale running job records
today, so there is nothing to demonstrate against either way; the reaper is project 93.
Rolling back a bad image is a different incident¶
If you have pinned a bad image and not lost a host, you are not on this page. See Roll the fleet back.