Skip to content

Stand the health worker down

Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: none, retired with Core.

An always-on monitor host sweeps the fleet, and the Cloudflare Worker should stop doing the same work beside it. The cutover is one variable and one deploy. Fleet owns the standby gate; this page is the act of arming it.

How you know you are in it

  • A monitor host exists, runs continuously, and publishes a state document on its own tunnel hostname.
  • MONITOR_STATE_URL is still commented out in worker/wrangler.toml. Unset means fail-active, and that is the fleet's posture for as long as no such host exists, so an unset variable is a setting rather than an oversight.
  • The shadow week has passed. See the prerequisite below; it is not optional and it is not a formality.

If no monitor host exists, stop here. A gate pointed at a document nobody maintains stands the Worker down for as long as that document keeps looking fresh, and nothing sweeps the fleet in the meantime.

The prerequisite: seven days of agreement

The Worker is the fallback, and a fallback that disagrees with the primary is worse than no fallback. So before the gate is armed, both sweeps run against the same fleet for a week and are compared — hourly, by the shadow:observe schedule on the fleet contract repository.

A week rather than a run, because the interesting states are rare: a host going stale, a digest mismatch after a deploy, a takeover arming, the daily summary rolling over at midnight UTC. A single comparison exercises none of them and proves only that two processes can agree about a quiet fleet.

Two things decide whether the week passed, and they are separate questions:

node restoration/shadow.mjs report --ledger restoration/shadow/ledger.jsonl
node restoration/shadow.mjs gate   --ledger restoration/shadow/ledger.jsonl
  1. Did both sides actually sweep? Asserted through signals a conditional write cannot suppress — the monitor's sweptUtc, which advances on every completed sweep, and the Worker's KV read count, because reads track sweeps while writes track changes. A row whose two documents agree while either side did not sweep is recorded as inconclusive, never as agreement. Two silent processes agree perfectly.
  2. Did they conclude the same things? Compared with comparableStateDoc, the same function the Worker's own conditional write consults, so there is one definition of "a difference" rather than two. It drops the fields two observers differ on by construction: updatedUtc, takeover.lastSeenUpUtc, and per-host avail and syncAgeMin. All four are clocks.

A residual difference is a finding to explain, not a threshold to tune. gate refuses until each differing path has been named as explained; it will not average them away.

The disagreement to expect first is the monitor being right

The Worker's expectations come from a KV projection that fleet:provision writes; the monitor's come from fleet/hosts.yaml in git. When a provision run fails, KV keeps the old pin and the Worker keeps watching a roster that no longer exists — silently. That was the live state on 2026-08-20, and it shows up as differing image and hosts.*.skewed paths. Removing that failure mode is the point of reading the authority instead of the projection.

Nerthus.Infra.Restoration is the repository that stands the monitor host up (Install-NerthusRestoration.ps1; the dev monitor publishes its state document on dev-restoration.nerthus.pl). A fresh install runs in the shadow posture — it sweeps and publishes, and posts nothing — until the monitor's env file is written by hand, and the gate is armed only after a shadow week.

The monitor's sweep itself ships in the fleet contract under restoration/: the Worker's fleet-health.js run off Cloudflare through a file-backed FLEET_KV shim, reading the roster from git rather than from KV, sweeping every NERTHUS_RESTORATION_SWEEP_MINUTES (default 15) and publishing the state document atomically for the Worker's gate to read. The box's hostname restoration is a reserved service name no town slug may take: the box holds no lore and is never a takeover candidate.

The cutover

  1. Publish the monitor's state document on the monitor host's own tunnel hostname. The probe is a plain HTTPS GET with a five-second timeout, and it reads two fields: sweptUtc, the ISO-8601 UTC instant of the monitor's last sweep, rewritten every sweep, and activeSinceUtc, when the monitor's current run began. A non-200, a timeout, a body the Worker cannot parse, and a sweptUtc 40 minutes old or older each leave the Worker sweeping.

  2. Point the variable at that URL under [vars] in worker/wrangler.toml, then deploy from the fleet contract checkout:

    wrangler deploy -c worker/wrangler.toml

    Never the apex

    The target is the monitor host's own hostname, never the apex (evocation.nerthus.pl). The apex is the record failover moves, so a probe bound to it stops finding the monitor after exactly the event the gate exists for, and the Worker goes active for good.

  3. Confirm the flip. Each standby cron logs one standby event in the Worker's tail, and the active-to-standby edge posts one line to kanal-wewnetrzny-it: Flota <env>: Monitor aktywny - Worker w trybie czuwania (0 operacji KV). The Worker records that edge only while the monitor's activeSinceUtc is under 30 minutes old, so deploy against a young monitor run or expect the notice after the monitor's next restart.

  4. Hand the sweep back by commenting the variable out and deploying again.

What standing down costs

In standby the Worker runs no sweep. The host probes, the digest and schema comparison, the transition lines, the daily summary and the takeover arming all stop with it, and the monitor host owns every one of them.

That is one switch and not two, and it is a decision rather than a mechanism. The takeover leg needs GL_FLEET_TOKEN to open and auto-merge the role-flip merge request, and that token is on the monitor box as well as in the Worker. Had it been withheld from the box, "standby" would have had to mean something narrower — the sweep stopping while the takeover kept running, which needs the host probes that decide when to take over, which is the sweep. There is no such narrower sense here.

The Worker becomes the dead-man watcher underneath: a sweptUtc that ages past 40 minutes, two missed monitor sweeps, puts it back to sweeping on its very next cron and posts the resume notice (Manage notifications holds the catalogue). Worst case is 40 minutes of hysteresis plus one cron tick, so under 55 minutes.

Against the free tier, the two modes are ~96 KV writes a day against zero (Check the quota owns which column applies to a given month).

A monitor that publishes while its sweep has stopped would keep the Worker down

The gate keys on freshness, so a monitor that dies mid-sweep leaves a document that ages out and the Worker resumes — that is the design and it works. The subtler failure is a monitor that keeps publishing with a dead sweep: the document would stay fresh and the Worker would stay down.

It cannot happen here, and the reason is one rule in publish.js: sweptUtc is stamped only by a sweep that completed. The publisher re-publishes after a failed sweep too, carrying the old value, so a run of failures ages the document exactly as a dead process would. One process stamps it, and it is the one whose liveness is in question.

See also