Bootstrap a fleet host¶

Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: Adopt a host.

This guide stands up one fleet host: a machine that runs nerthusd in a container behind an outbound-only tunnel, converges with origin, and answers at a per-town hostname. The Fleet page owns the model — roles, the converge loop, failover; this page is the procedure. Most hosts are contributor devices. One is a rented server, and the two shapes differ enough that the differences are named below rather than left to the reader.

Prerequisites¶

• Docker and the Compose plugin. The whole host is two containers under docker compose.
• A modest machine: 2 cores, 4 GB RAM, 3 GB free disk. The daemon and cloudflared run inside the compose limits (3072M + 128M).
• Outbound 443 only. The tunnel dials out; the host opens no inbound port, ever.
• A platform the pinned digest carries. versions.lock pins an image by digest and Docker selects from it by host platform. Measured against the registry on 2026-08-03, that digest is an OCI index carrying both linux/amd64 and linux/arm64. The question a refused pull answers is whether the index contains this host's platform: docker manifest inspect <ref> | jq '.manifests[].platform'.
• Membership. The host's owner must be a member of the lore project — provision re-checks it every run.

Supported platforms¶

Three platforms are supported, and the host boundary is always a real Linux with a real init:

• Linux with systemd — the native case.
• macOS with launchd — Docker runs in a Linux VM; a launchd job drives the converge timer.
• Windows via WSL2 — a systemd-enabled WSL2 distro is the device (see below).

Git Bash and native Windows are not supported. Git Bash mounts with noacl, so the chmod 600 the key posture depends on silently no-ops, and a Windows-native agent would fork the one contract into two implementations.

A contributor device, or a rented server¶

The three platforms above are init systems. They say nothing about who holds the disk, and that is the other axis a host sits on. Both shapes run the same agent, the same two containers and the same converge loop; the procedure below is the same procedure. What differs is the first hour and what counts as normal afterwards.

A rented server reaches step 1 only after Commission a rented host, which owns the credential handling and the hardening. Running bootstrap.sh first leaves a mailed root password guarding a public IP with password authentication on and no firewall, and the fleet's device keys land on that disk a minute later.

Fleet owns where lore is allowed to rest. A rented host holds the same corpora every other host clones — the campaign, the moderation corpus, the transcript archive, the annotation tree — so the residency question is answered before the roster line is written, not after.

One machine, several towns¶

Everything below installs one town per machine: a root-owned checkout at /opt/nerthus/infra, keys under /etc/nerthus, and a single system unit. That suits a device that is a fleet host.

The dev fleet is not shaped that way, and the pages that describe only the shape above describe a machine this estate does not run. Its towns are compose stacks on one laptop, and each gets:

• a user-domain launch unit of its own — ~/Library/LaunchAgents/pl.nerthus.agent.<town>.plist on macOS, StartInterval 300 — because a single root daemon can converge one town and this machine converges several;
• its own NERTHUS_CONF_DIR (~/.nerthus/fleet-conf/<town>, holding that town's age key and deploy key) and NERTHUS_STATE_DIR (~/.nerthus/state/<town>, holding its rendered .env);
• one shared NERTHUS_INFRA_DIR at ~/.nerthus/infra. The agent only ever hard-resets that checkout to origin, so sharing it is safe — and the agent serialises the fetch/reset/submodule sequence behind a lock, because several agents taking .git/index.lock on one five-minute interval means the loser converges nothing and reports it as an exit code nothing reads.

Nothing needs root in that arrangement: Docker Desktop reaches the daemon as the logged-in user, and every path is under that user's home. bootstrap.sh does not build it — the units and directories are made by hand, and each town's roster line still comes from a bootstrap run or from the age key it generated.

The cost is stated rather than discovered: every town on that machine shares one Docker daemon, one disk and one set of egress, so it is a development arrangement and not a residency model. Fleet owns where lore is allowed to rest.

Bootstrap the host¶

1. Clone the fleet repository with submodules into /opt/nerthus/infra on the device. Which repository is an input to this runbook, not a default, because the estate has two fleets and only one of them exists today:

sudo git clone --recurse-submodules https://gitlab.com/margonem/nerthus/it/nerthus.core.infra.dev.git /opt/nerthus/infra

The command uses the development row, because it is the only one that clones today; substitute the production repository the day it exists. This line used to name production unconditionally, which made step 6 of the Hosting reading order fail at its first command - and on_undefined cannot catch that, because the variable resolves. It just names nothing.

The path is the contract. The converge agent reads NERTHUS_INFRA_DIR and falls back to /opt/nerthus/infra, and neither the systemd unit nor the launchd job exports that variable. A clone anywhere else leaves every tick dying at exit 0 with no infra checkout at /opt/nerthus/infra: a host that never converges and never complains. To keep the checkout somewhere else, export the variable in the unit yourself — a systemd drop-in carrying Environment=NERTHUS_INFRA_DIR=…, or the plist's EnvironmentVariables dict on macOS.

1. Run the bootstrap script as root with your town slug. It checks Docker, Compose, age, and git; generates the device's age key and lore deploy key under /etc/nerthus/; installs the converge agent and its timer (or the launchd job); and prints one ready-to-paste roster line.

sudo /opt/nerthus/infra/common/bootstrap/bootstrap.sh torneg

Root is required: the script writes /etc/nerthus and installs a system unit, so an unprefixed run exits 2 before it does anything. On an unsupported host it exits 3 with a named reason before creating any key or unit, and prints no roster line.

Read the checkout line it prints above the roster line. [bootstrap] infra checkout: /opt/nerthus/infra means step 1 landed where the agent will look. A NOTE: no infra checkout at /opt/nerthus/infra means it did not. The script carries on either way, and it never adopts the directory you ran it from.

1. Paste the printed line into fleet/hosts.yaml and open a merge request. The printed line always says role: replica, so change that word on the first host you bootstrap: that one is the primary. An absent takeover: field reads as manual, and auto is the automatic-failover opt-in (Run a failover):

torneg: { owner: anward, role: primary, takeover: manual, age: age1ga25gx…kuk, key: "ssh-ed25519 AAAAC3Nza…GKWv nerthus-torneg" }

1. Merge is the approval. On merge, the fleet pipeline validates the roster and provisions Cloudflare: the named tunnel, the proxied CNAME with a collision read-back, the apex pointer if this line is the primary, the edge rules, and the age-encrypted tunnel token committed back to the repo.

2. Install the deploy key by hand. Paste the roster line's key: value into the lore project's deploy keys, read-only. Provision never installs a key; it reconciles the can_push bit of one already there, and the primary's is the one key it raises. A host whose key was never pasted gets a single provision log line — deploy key for '<town>' not installed on the lore project — the sweep already names this; skipping its write bit — and a lore clone that never works.

When the fleet's settings carry mc_repo_url, the same keys must also be enabled on the moderation project, or no host can clone that corpus. Provision does this itself — deploy_keys/:id/enable, read-only, never raised — and reports per key; a run where zero keys enabled and some failed reports the corpus as staying unreadable. The hand remedy on a 400 is a Maintainer on the moderation project enabling the privately-accessible key under Settings > Repository > Deploy keys, read-only.

1. Wait for convergence. Within about five minutes the agent decrypts its token and brings the stack up; the health worker sweeps every fifteen minutes, so its convergence tick reaches the tech channel within about fifteen minutes of the host first answering on the pinned image. Silence before then says nothing.

The role model¶

Exactly one host is primary — the first one bootstrapped — and it is the only host that publishes to origin. Every other host is a replica: it boots read-only and converge-only, serving current data without writing. Roles live in fleet/hosts.yaml, one line per host, and a role change is a one-line merge request. When the primary goes dark, automatic failover can promote an opted-in replica; a hand-edited role flip is the fast path when someone is at the keyboard.

What normal looks like¶

• A sleeping contributor device is fine. Its hostname returns a 530 at the edge and the health sweep names it; overnight for a laptop, that is expected.
• The same 530 from a rented server is an outage. That host has no sleep state, so the alert names the tunnel, the container or the machine, and one of them needs a human. Read the rule by host shape, not by the alert text — the two look identical at the edge.
• A post-sync pause is a rebuild, not a hang. After a converge pulls model-relevant bytes, the daemon rebuilds its in-memory model before serving the next request. A brief stall there is the self-heal working.

Offboarding¶

To remove a host, run the uninstall script inside its environment, then drop its line from fleet/hosts.yaml:

sudo /opt/nerthus/infra/common/bootstrap/uninstall.sh --purge

Uninstall stops the schedule, and --purge deletes the device keys and the rendered .env that held the plaintext tunnel token. The lore volume and the containers survive it, so the connector stays up until somebody brings the stack down under the town's own compose project. Offboard a host is the full procedure, including that teardown and the Cloudflare cleanup.

See also¶

• Commission a rented host — the first hour on a machine that arrives with a mailed root password
• Fleet — the model behind every step here
• Publish the API — the edge the fleet sits behind
• Sync — what a host does on every converge tick
• Fleet runbooks — failover, diverged recovery, rotation, WSL2, quota, offboarding