Skip to content

Offboard a host

Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: Retire a host.

Remove a host cleanly when a contributor leaves, a device is retired, or a host flaps so persistently it costs more than it gives. The order below is load-bearing: Cloudflare refuses to delete a tunnel that still has a live connector, so the device stops first. Fleet owns the roster and data-residency model.

Order

  1. On the device (the owner, or Rada if they still hold it), purge the install:
sudo /opt/nerthus/infra/common/bootstrap/uninstall.sh --purge

This stops the schedule so nothing converges the stack back up, and --purge deletes the device keys and the rendered .env that held the plaintext tunnel token. Without --purge the keys survive, which is right for a pause and wrong for an offboarding.

A machine that hosts several towns has one schedule and one key directory PER TOWN, and the script above removes neither. That is the dev fleet's arrangement — every town is a compose stack on one laptop, each with its own launch unit, CONF_DIR and STATE_DIR — so on such a machine the schedule and the keys come off by hand, per town:

# macOS: one user LaunchAgent per town
launchctl bootout gui/$(id -u)/pl.nerthus.agent.<town>
rm ~/Library/LaunchAgents/pl.nerthus.agent.<town>.plist

# the town's own key directory and rendered .env
rm -rf ~/.nerthus/fleet-conf/<town> ~/.nerthus/state/<town>

Leave the shared ~/.nerthus/infra checkout alone — the remaining towns converge through it.

Check what is still loaded before you call this done. An agent left ticking for an offboarded town fetches the infra repo every five minutes, finds no descriptor for a town nobody provisions, logs that it found none, and races the towns that are still real for the shared checkout's index.lock:

launchctl list | grep nerthus     # only the towns still in the roster
ls ~/.nerthus/fleet-conf          # same list
  1. Bring the stack down under the town's own compose project, and take the volume with it. The agent created the stack as project nerthus-<town>. The uninstall script tears down a project named nerthus instead and swallows the miss, so it prints its success line while the containers and the lore volume are still there:
docker compose --project-name nerthus-torneg down --volumes
docker volume ls    # nerthus-torneg_lore must be gone

Read the listing. Two things depend on this step: the lore actually leaving the device, and the connector stopping, without which step 4 cannot delete the tunnel.

  1. Remove the line from fleet/hosts.yaml in a merge request, and merge it. Provision is additive and deletes nothing, so after this merge the host is simply no longer desired.

  2. Run the manual fleet:decommission job on the default branch. It deletes that host's tunnel, its DNS record, its worker route, its secrets/<town>.tunnel.age, and its rendered/<town>.json, then commits the ledger back.

  3. Remove the deploy key from EVERY corpus repository, not only the lore one. One key per town is shared onto each repository the fleet reads — the lore tree, the moderation corpus, the transcript archive, the annotation tree, and any other remote schema/remotes.yml declares — so removing it in one place ends access to one repository and leaves the rest. Sweep by title, which is nerthus-<town>-r2:

for p in repozytorium-fabularne-prodkopia repozytorium-moderatorskie-dev \
         repozytorium-transkryptow-dev repozytorium-lingwistyczne-dev; do
  glab api "projects/margonem%2Fnerthus%2Fit%2F$p/deploy_keys"
done

Delete every entry titled for the departing town, then re-run the listing and read it. This is the step that ends the device's access; do not skip it because the volume was already wiped. fleet:provision will not undo it — it is additive and shares keys for towns the roster still names.

  1. Confirm, by reading rather than by assuming. The hostname stops resolving and the next sweep no longer lists the host — and the deploy-key listing of every corpus repository equals the roster, and launchctl list | grep nerthus equals the roster too. An offboarding that removed the roster line and left the machine and the grants behind reads exactly like a complete one from the roster's side.

If the device is unreachable

A lost or bricked device runs neither step 1 nor step 2, so its connector may still be alive. Do steps 3–5 anyway, in that order, and expect fleet:decommission to fail on the tunnel delete while a connector is registered. Then:

  1. Rotate that host's tunnel token first with fleet:rotate; the running connector is invalidated at once (Rotate secrets).
  2. Wait for Cloudflare to drop the stale connector, then re-run fleet:decommission.

Treat a lost device as a credential incident, not only a hardware loss: the deploy key and the last decrypted tunnel token were on it, and that key opened every corpus repository rather than only the lore one. Removing it everywhere, per step 5, is what closes it.

Chronic flapping

A host that oscillates between up and down posts a Discord transition every sweep, and the tech channel fills with up-and-down lines until the notices that matter are lost among them. Ask the owner to fix the device or to leave the fleet; if neither happens, remove the line. This is a social decision on a technical trigger.

Data residency

A completed offboarding is what makes git log fleet/hosts.yaml true again as the residency register, which is why step 2's volume removal and step 5's key removal are not optional housekeeping. Fleet owns the residency model.

See also

  • Fleet — the roster and where lore is allowed to rest
  • Rotate secrets — the tunnel-token rotation a lost device forces
  • Check the quota — the monthly reading of the free-tier budgets