Read the logs¶

Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: not yet written.

A daemon that answers nothing, a converge that never lands, a Discord line that never arrives: the first question is always where does this write down what it did. It writes down a great deal, and almost none of it is where a person looks first. This page is the map, and the one thing worth knowing before anything else is the negative:

The one line is worth reading once, because it is the only thing that surface ever says. On a primary it is SyncStale: last successful converge never …; on a replica it is ReadOnly: daemon is in read-only mode. Both come from Update-NerthusModel at boot and neither is re-emitted.

The five surfaces¶

Every figure below was taken on 2026-08-28 on the host named beside it.

And what each held when this page was written:

A replica has no audit.jsonl at all. Only torneg carries one, because only a primary writes, and the audit stream records writes. Looking for an audit trail on a replica and finding no file is the expected answer rather than a missing one.

Where they live, and what destroys them¶

/repo/.nerthus/log/ is inside the lore volume, and .nerthus/.gitignore ignores it. So the largest and most useful stream on any host is untracked, unpushed, and lives in the one directory a docker compose down -v or a clone-and-recreate throws away. Nothing backs it up and nothing warns you.

Copy before you recreate. One command, before the stack comes down:

docker cp nerthus-torneg-nerthusd-1:/repo/.nerthus/log ./log-torneg-$(date +%F)

request.jsonl and operational.jsonl are the two worth the disk: the first is what the host was asked, the second is what it decided. events.jsonl is replayable from the corpus; audit.jsonl is not, and it is the smallest.

Nothing rotates. No stream, on any host, has a rotation policy — not the JSONL files, not the laptop's agent logs, not frog01's monitor log. eder's agent journal is the exception, because journald applies its own limits. The streams are small enough that this has never mattered (request.jsonl, the largest anywhere, was 7.6 MB after six days) and it is stated here so that nobody discovers it as a surprise.

Reading a stream¶

Each line is one JSON object with a ts. The fields differ per stream, and knowing them is most of the work:

Four filters that answer the four questions people actually arrive with. Each runs inside the container, so jq is not required on your machine:

C=nerthus-torneg-nerthusd-1

# Is anything failing? Every non-2xx the daemon answered, newest last.
docker exec $C sh -c 'grep -v "\"status\":2" /repo/.nerthus/log/request.jsonl | tail -20'

# Did the converge tick run, and what did it do?
docker exec $C sh -c 'grep "\"comp\":\"sync\"" /repo/.nerthus/log/operational.jsonl | tail -10'

# Did a Discord message go? A SKIP carries its reason.
docker exec $C sh -c 'tail -20 /repo/.nerthus/log/deliveries.jsonl'

# Who wrote to this host, and to what?
docker exec $C sh -c 'tail -20 /repo/.nerthus/log/audit.jsonl'

A real deliveries line, from torneg on 2026-08-23:

{"entity":"","channel":"kanal-wewnetrzny-it/it","reason":"fetch_failed","result":"SKIP","operation":"peer-watch","ts":"2026-08-23T19:15:05.8681117Z"}

result: SKIP with reason: fetch_failed is a message that was composed and never sent. Nothing else on any surface says so — the channel simply stays quiet, which looks exactly like nothing having happened. That is the single most useful reason to know this stream exists.

The converge agent is a different log, on a different clock¶

The daemon's streams say nothing about whether the host is converging: that is the agent, and it runs outside the container.

On the laptop each town has its own file and none of the lines carry a timestamp:

tail -20 ~/.nerthus/log/agent-torneg.log

You get [nerthus-agent] converged and no time, so the only way to date a tick is the file's own mtime and the order of the blocks. Two of the four files on that machine — agent-tuzmer.log and agent-werbin.log — stop on 2026-08-21 and end with no rendered/<town>.json yet — the fleet pipeline has not provisioned, which is what a town that was never finished looks like from here.

On eder the same agent runs as a systemd one-shot behind a timer, so the journal carries it with timestamps and it is the better surface by a wide margin:

journalctl -u nerthus-agent.service -n 40 -o short-iso
systemctl list-timers nerthus-agent.timer

The service reads inactive between ticks and that is correct — it is a one-shot. The timer is the thing that should read active.

The monitor writes somewhere else again¶

frog01 runs the always-on fleet-health sweep as an OpenRC service, and writes one JSON object per sweep:

ssh -p 11190 frog@frog01.mikr.us 'rc-service nerthus-restoration status; tail -3 /var/log/nerthus-restoration.log'

Each sweep object carries utc, durationMs, peakRssMb and the roster it read. The two watchers is the model behind it, and Tell which watcher is live is the page for the first minute of an incident.

What this page does not cover¶

• Game transcripts. nerthus.logs/ is the archive of session logs and has nothing to do with these streams; Session logs owns it.
• The Discord alert channel. What a kanal-wewnetrzny-it line means is Read the fleet alerts. This page is where you go when the channel said nothing.
• CI job logs. They live in GitLab and evaporate with the job; Set up pipelines says which artefacts survive.

See also¶

• Read the fleet alerts — what the channel says, when it says anything
• The two watchers — which sweep is live, and where each one writes
• Recover a diverged host · Recover a failed commit-back
• Sync & deployment — the converge tick these logs record