Tell which watcher is live¶

Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: none, retired with Core.

Two things can be sweeping the fleet (The two watchers). This page is how you find out which, in the first minute of an incident, and how to hand the sweep back on purpose.

From the panel¶

Open Flota. The Czuwanie pane in the right-hand column answers directly:

Czuwa: Monitor (host własny) — the monitor swept recently and the Worker is standing down.
Czuwa: Worker (Cloudflare) — the monitor is unreachable, its document is unreadable, or its last sweep is 40 minutes old or older, so the Worker has resumed.
Czuwa: Nieznane — no document at all.

Under it: the age of the last sweep, which producer wrote the document, and a reminder that the monitor's home is on a deletion clock.

The pane fails towards the Worker, never towards the monitor. Every unclear answer reads as Worker, because that is what the Worker's own gate does with the same input — a panel that guessed "monitor" on a malformed document would tell you the fleet was watched by something that had stopped.

The panel needs the monitor to allow its origin

The pane reads the document from the browser, so the monitor must name the panel's origin in NERTHUS_RESTORATION_STATE_ALLOW_ORIGIN. Unset, the pane says brak odpowiedzi lub CORS — and the same document still answers curl perfectly. If the panel disagrees with the command line below, suspect the header before you suspect the box.

A dashboard with no monitor configured for its hostname says so in the pane rather than showing a blank. ?monitor=<url> points it at one for the length of a page load, without persisting.

From the outside, with `curl`¶

The monitor's document carries its own producer and its own age:

curl -s https://dev-restoration.nerthus.pl/state.json | jq '{producer, sweptUtc, activeSinceUtc, roster}'

{
  "producer": "monitor",
  "sweptUtc": "2026-08-20T21:20:55.636Z",
  "activeSinceUtc": "2026-08-20T21:05:54.033Z",
  "roster": { "stale": false, "error": null, "head": "a012733020a9ac3f386744a56deb19e91464f40b", "staleForMs": 0 }
}

producer — who wrote this document. This is the field to read, not workerMode.
sweptUtc — the last sweep that completed. Under 40 minutes old and the Worker stands down.
activeSinceUtc — when the monitor's current run began, not when the document was written.

roster — the state of the monitor's own input, and the reason it is here rather than only in the log: through the sweep, a roster that was never cloned, one whose fetch failed, and one the parser refuses are indistinguishable — all three read as an absent hosts.json. Those are three different problems in three different places.

what you see	what it means
`stale: false`, `head` set	the roster is current, at that fleet-contract commit
`error` naming a path under the state dir	nothing has been cloned yet
`error` starting `fleet clone fetch:`	the tree is on disk and behind origin; `staleForMs` says how far
`error` naming a `hosts.yaml` line	the roster is there and the parser refuses it

head is the answer to "which roster is it watching", which the KV projection could never give.

workerMode does not mean what its name suggests on this document

The monitor runs the same code the Worker does, finds no MONITOR_STATE_URL, sweeps, and stamps workerMode: "active" into its own document. Read as "the Cloudflare Worker is sweeping", that field tells you the opposite of the truth on the one document that proves the Worker is standing down. producer is the field that carries this meaning.

And the Worker's own view, which needs no credential:

curl -s https://dev-fleet-health.nerthus.pl/ | jq '{updatedUtc, workerMode}'

On the Worker's document workerMode does mean what it says: active while it is sweeping, standby from the moment it first stood down.

This used to name fleet-health.nerthus.workers.dev, and that hostname is gone

The Worker answered on two hostnames, and the workers.dev one was an oversight rather than a policy: it is not in zone nerthus.pl, so it sat outside the zone's rate-limit ruleset, its managed WAF and its cache rules, while the account's other two Workers already had it disabled. Project 91 turned it off (workers_dev = false in worker/wrangler.toml, so a redeploy cannot bring it back).

dev-fleet-health.nerthus.pl serves the identical document — measured 2026-08-24T21:00Z, both hostnames 200 with the same 2 011 bytes. Only the address changes.

The call is still anonymous and still needs no credential. That is deliberate: two runbooks and one project's evidence rest on it, and closing it without a replacement instrument would leave this estate with fewer ways to see a wedged failover than it had before. What did change is the method — anything but GET and HEAD now answers 405, because POST / used to return the state document to anyone who asked.

What the HTTP status alone tells you¶

The monitor's hostname is served by a cloudflared tunnel, and three failures look the same from a browser and have fixes in three different places:

status	what it means	where to look
`200`	the monitor is answering	—
`502`	the tunnel's ingress rule is correct and nothing is listening on `:8788`	the `nerthus-restoration` service is down
`503`	the connector has no ingress rule	`~/.cloudflared/config.yml` never landed
`530`	Cloudflare has the DNS record and no connector is registered	the `nerthus-restoration-tunnel` service is down

On the box¶

ssh -p 11190 frog@frog01.mikr.us 'rc-status | grep nerthus; tail -3 /var/log/nerthus-restoration.log'

One JSON line per sweep:

{"event":"sweep","utc":"2026-08-20T21:20:55.636Z","durationMs":421,"peakRssMb":74.9,
 "roster":{"stale":false,"error":null,"head":"a012733020a9ac3f386744a56deb19e91464f40b"}}

roster.head is the fleet-contract commit the sweep's expectations came from — the answer to "which roster is it watching", which the KV projection could never give.

A clone-failed line means the roster fetch failed; the sweep continues against the last good copy and marks it stale, so a broken clone degrades visibly instead of silently.

Hand the sweep back deliberately¶

Comment MONITOR_STATE_URL out under [vars] in worker/wrangler.toml and deploy from a fleet-contract checkout:

wrangler deploy -c worker/wrangler.toml

How long the fleet is unwatched: at most one cron tick, so under 15 minutes, and usually zero. The Worker goes active on its very next scheduled run after the deploy, and the monitor does not stop — it keeps sweeping and publishing, so for that window both are watching and nothing is unwatched at all. This is the safe direction.

The reverse — arming the gate — belongs to Stand the health worker down, and it is the one with a prerequisite.

If the monitor dies rather than being stood down¶

Nothing to do: the Worker notices by itself. sweptUtc ages past 40 minutes, the next cron sees a stale document, and the Worker resumes sweeping and posts its resume notice to kanal-wewnetrzny-it. Worst case is 40 minutes of gate hysteresis plus one cron tick, so under 55 minutes.