Read a dead-man alert¶

A monitor host runs a dead-man: a short shell script, on a box that holds no lore, answers no read and carries no site role. It asks whether the site still answers and posts to Discord when that changes. It depends on neither the site nor the bot, which is the whole reason it exists — an all-dark estate has to have one path left that can still tell a person.

The three messages¶

The heartbeat is the instrument, not decoration. An alert path that has never delivered looks exactly like a quiet estate, and this estate has shipped precisely that: the old health Worker threw away Discord's response, and nobody knew for weeks. So the dead-man posts once a day, and it reads the status of every post it makes — a delivery failure is written to its log with the code.

Silence is a finding. If no heartbeat arrived yesterday, the dead-man itself is what is wrong. Start with the box, not with the site.

When a target does not answer¶

The status code in the message is the first thing to read, and it distinguishes failures that look identical from a browser:

The 502/503 pair was measured on the live box and sends you to two different places, so read the code before anything else.

On the box¶

cat /etc/nerthus/deadman.conf          # what it watches, where its state is; no secret is in it
ls /var/lib/nerthus/deadman/           # one failure count per target, an alert flag, the heartbeat day
tail /var/log/nerthus-deadman.log      # every sweep, and the status of every post

State is deliberately not on tmpfs: a reboot would forget which alerts were outstanding and re-announce every target the moment the box came back.

What it cannot tell you¶

That the webhook still reaches a channel somebody reads. Using a webhook posts a message, so a probe would write into the channel every time the estate asked whether the credential was alive — the daily heartbeat is that check, which is why the credential's row records no other one.

Two dead-men¶

The estate runs one on frog01 (Mikrus) and one on nerthus-host-8 (netcup): different providers, different networks, so a site is never watched only from its hub's provider. If exactly one goes quiet, suspect that box; if both do, suspect the thing they share, which is Discord.