Recover a diverged host¶

Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: not yet written.

Before you start: where the evidence is

The daemon writes almost nothing to docker logs and a great deal to its own JSONL streams inside the lore volume. Read the logs is the map, and the converge tick this page is about is recorded in operational.jsonl and in the agent's own log, not in the container's stdout.

Diagnose a host the sweep names as reachable but stale, or one that never appeared, and put it back in step. Fleet owns the converge loop; this page is the fixes.

Read the fleet first¶

One command, from a checkout of the fleet repository, prints what every source says and where they disagree:

pwsh common/ci/Get-NerthusFleetState.ps1 -Repo .
pwsh common/ci/Get-NerthusFleetState.ps1 -Repo . -Token $IT_TOKEN   # + the commit, the clone, the corpus

Secretless by default — the two committed documents, each host's anonymous /health and /fleet/status, and the health Worker's public GET. -Token (a token carrying the it bundle) adds /sync and /logs/status, which is what turns the diagnosis below into a column: diverged is ahead > 0, dirty is a modified file in the serving clone, lore-divergent is a host serving a commit the fleet does not, and fetch-stale is a converge that looks fresh because the fetch underneath it stopped working. -Offline compares the two git documents alone, which is what a merge request wants. Exit 1 means at least one finding; -Format json for anything that is not a terminal.

-GitLab adds the control plane — a failed provision, a provision with no rotation after it (ciphertext-suspect), an open pin or takeover merge request, a roster key installed on no lore deploy key (key-missing, the commonest first-join failure), and the month's CI minutes. It needs a read_api PAT in NERTHUS_GL_READ_TOKEN, and reads only.

To watch rather than sample:

pwsh common/ci/Get-NerthusFleetState.ps1 -Repo . -Token $IT_TOKEN -Follow

One NDJSON line per event across every host, resuming from a cursor per town. Pipe it to jq, a file, or an agent.

The Worker's public GET is the one input the checkout does not carry. It answers on https://dev-fleet-health.nerthus.pl/; pass it as -HealthWorkerUrl, export it as NERTHUS_FLEET_HEALTH_URL, or commit it as health_worker_url in fleet.settings.yml. Neither fleet commits that key, so a first run warns no health Worker URL … the swept column is unavailable and answers every other column.

This paragraph used to say the Worker answers on its workers.dev address

It was right about the config and wrong about the estate: worker/wrangler.toml still declares no route, but dev-fleet-health.nerthus.pl is a route created through the Cloudflare API and therefore invisible to wrangler. Project 91 disabled the workers.dev subdomain, so that address no longer answers and this one does.

The verdicts are one vocabulary across every surface — the table, the JSON, the journal — so skew-image here is skew-image there.

The stack is named after the town

The agent creates the stack as compose project nerthus-<town>, because one machine can hold several towns: the dev fleet runs four stacks on one Mac. So on torneg the daemon container is nerthus-torneg-nerthusd-1, the connector is nerthus-torneg-cloudflared-1, and the lore volume is nerthus-torneg_lore. Every command on this page spells the town out. A bare nerthus project matches nothing, and docker compose reports success on a project it did not find.

What the sweep says¶

What the sweep says	What it means
`Host <town>: Brak odpowiedzi`	No connector. The device is asleep, off, or the stack is down.
`Host <town>: Brak aktualizacji zmian od N minut`	Container up, sync tick failing, usually git.
`Host <town>: Obraz Docker nieaktualny`	The agent is not converging, or the upgrade wave is holding it back on purpose.
No `Zaktualizowany` after a merge, ever	Almost always the deploy key was never installed.

The deploy key was never installed¶

This is the commonest first-join failure, and it looks like nothing happening at all. The device's first clone fails, so there is no repository and no sync.

Take the key: value from that host's merged fleet/hosts.yaml line.
Add it to the lore repository's deploy keys, read-only. (Write is enabled only for a contributor primary, and even then merge requests via the bot are the cleaner path.)
On the device, run docker restart nerthus-torneg-nerthusd-1, or wait: the entrypoint retries the clone on the next container start.

The agent is not running¶

On the device:

systemctl status nerthus-agent.timer     # Linux
launchctl print system/pl.nerthus.agent  # macOS
sudo /usr/local/bin/nerthus-agent.sh     # run it in the foreground and read the output

The agent exits 0 on every ordinary refusal and names it in one line: no rendered descriptor (the pipeline has not provisioned this host yet), no age secret (provision has not encrypted a token for this device's key), a contract major it does not support (re-run bootstrap.sh to reinstall the agent), or a fetch failure (the device is offline).

The clone diverged¶

The daemon converges fast-forward only, so a clone with local commits or rewritten history stops moving rather than merging blindly. On the device:

docker exec -it nerthus-torneg-nerthusd-1 git -C /repo status
docker exec -it nerthus-torneg-nerthusd-1 git -C /repo log --oneline origin/main..HEAD

If those local commits are already on origin (the usual case, a landing that pushed and then lost track), discard them:

docker exec -it nerthus-torneg-nerthusd-1 git -C /repo fetch origin
docker exec -it nerthus-torneg-nerthusd-1 git -C /repo reset --hard origin/main

If they are NOT on origin, they are unpublished work. Push them as a branch and open a merge request before you reset anything: the committed artifacts are the truth, and this is the one moment a device holds some that origin does not.

Nuclear option¶

The lore volume is a cache, not a source. Wiping it costs one clone:

sudo /usr/local/bin/nerthus-agent.sh                        # confirm the descriptor and secret are present first
docker compose --project-name nerthus-torneg down --volumes
docker volume ls                                            # nerthus-torneg_lore must be gone
sudo /usr/local/bin/nerthus-agent.sh                        # re-clone from scratch

Read the volume listing before you re-run the agent: it is the only evidence the wipe happened.

Gitignored private state on that host (tokens, grants) is lost with the volume and must be re-issued. On a replica that costs nothing. On the primary, read Run a failover first.

Image skew that is not a fault¶

A replica running a newer digest than the primary is the upgrade wave working as designed: replicas take a new image first and soak it while the primary waits. The sweep reports the skew because it cannot tell intent from fault. Fleet owns the wave; check it before treating skew as a problem.