Skip to content

The two watchers

Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: none, retired with Core.

Two things watch the fleet: a monitor on a box we do not own, and the health Worker on Cloudflare behind a standby gate. With one watcher the question "who is watching?" did not exist. With two it is the first thing anybody asks during an incident, so this page answers it, and Tell which watcher is live is how you check rather than assume.

Fleet owns the sweep itself — what it probes, what it decides, what it posts. This page owns only the fact that there are two of them.

The two, side by side

The monitor The health Worker
Where it runs frog01.mikr.us, a free Mikrus FROG, as the nerthus-restoration OpenRC service Cloudflare's edge, as fleet-health
What it runs restoration/monitor.js — the Worker's own fleet-health.js, imported unmodified fleet-health.js
How often every 15 minutes, a resident timer every 15 minutes, */15 * * * *
Where its expectations come from fleet/hosts.yaml in the fleet contract repository, cloned anonymously over HTTPS hosts.json in Cloudflare KV, written by fleet:provision
Where its memory lives JSON files under ~/.local/state/nerthus Cloudflare KV
What it publishes https://dev-restoration.nerthus.pl/state.json its own GET / endpoint
KV cost none — it holds no Cloudflare credential ~192 reads and ~96 writes a day when active, zero in standby

It is one implementation, not two. The monitor imports the same fleet-health.js the Worker runs; a shim supplies the two things Cloudflare provides and Node does not (env.FLEET_KV over a directory of JSON files, and ctx.waitUntil). Two implementations of one sweep would have to agree forever, and this fleet has already produced one instrument that drifted from its own documentation.

The gate, and which way it fails

The Worker probes the monitor's state document before every sweep and stands down when it is fresh. Fresh means a sweptUtc under 40 minutes old — two missed 15-minute monitor sweeps, so a single late sweep does not flap the gate.

Everything else means the Worker sweeps:

  • the probe hostname is unset (this is the posture today, and it is a setting rather than an oversight),
  • the document is unreachable, a non-200, or unparsable,
  • sweptUtc is missing, unreadable, or 40 minutes old or older.

Fail-active is the rule. The only state that stands the Worker down is a monitor that has demonstrably swept recently; every ambiguity leaves two watchers rather than none.

sweptUtc is the last sweep that completed

A sweep that fails does not advance it. That is the field's whole point: a monitor failing every single sweep would otherwise look perfectly fresh forever, and the Worker — seeing a fresh document — would stay in standby and never take over. The dead-man would be inverted into a silent total failure.

So the freshness bound comes from the sweep, not from the publisher. The monitor re-publishes after a failed sweep too, carrying the old sweptUtc, which lets the document age until the Worker resumes.

What standing down stops — all of it

The runbook used to say the takeover arming stops with the sweep and that "the monitor host owns every one of them", and left open whether that could really be one switch. It can, and the reason is a decision rather than a mechanism: the monitor holds GL_FLEET_TOKEN as well as DISCORD_BOT_TOKEN, so it owns the role-flip merge request alongside the probing.

In standby the Worker therefore runs nothing: no host probes, no digest or schema comparison, no transition lines, no daily summary, no takeover arming, and no KV operations at all. There is no narrower sense of "standby" hiding in that sentence.

The monitor needs its actuation coordinates or the takeover leg refuses

apiBase and projectPath cannot come from git — fleet:provision writes them into KV from CI variables, and no committed file carries them. The monitor is told them through NERTHUS_RESTORATION_ACTUATION_*. All three or none: without them the takeover arming refuses by name (brak actuation.apiBase/projectPath w KV) rather than crashing, which is a failure you would meet at a failover and not one moment earlier.

The monitor's home is on a deletion clock

The box is deleted after three months without a login. That is the provider's rule, verbatim from https://frog.mikr.us: «Serwery, na które nikt nie logował się przez 3 miesiące, są usuwane.» The clock started 2026-08-04 and the assumed expiry is 2026-11-02 — ninety days, the shorter reading of "3 miesiące", so the error is early.

It is a free box, 256 MB with no swap, Alpine on LXC, not our hardware and with no backups. Putting the fleet's eyes on it made that clock load-bearing, and Keep the monitor box alive is what keeps it from running out.

Nothing on our side can confirm a login landed. /var/log/wtmp is zero bytes on that image and nothing writes to it, so last returns nothing and always will. A machine logs in monthly, and a human must log in too, because nobody can prove the machine's attempt counted.

What breaks if the box vanishes

Not much, and that is worth knowing before it happens rather than during. The monitor's state document stops being served, the Worker's gate stops seeing a fresh sweptUtc, and within 40 minutes the Worker resumes sweeping on its next cron and posts its resume notice. The fleet keeps being watched, with a 15-minute detection floor instead of a 15-minute detection floor — the two sweep at the same cadence, so nothing changes but who is doing it.

What is actually lost: the second vantage. The Worker sweeps from Cloudflare's edge and the monitor from a different network, so a host dark to one and bright to the other is a fault only the pair can separate. And the Worker starts spending KV again — about 192 reads and 96 writes a day, against a free cap of 1,000 writes (Check the quota).

That is an inconvenience, not an outage, and the difference is the whole reason the Worker stays deployed.

Which one is sweeping today

The Worker. MONITOR_STATE_URL is commented out in worker/wrangler.toml, so the gate is dormant and the Worker is active — with the monitor sweeping and publishing beside it, depended on by nothing.

That is the shadow week, and it is deliberate: the Worker is the fallback, and a fallback that disagrees with the primary is worse than no fallback at all. The two are compared hourly, and Stand the health worker down is the act that arms the gate once they have been shown to agree.

See also