The fleet¶

Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: none, retired with Core.

Nerthus.Core runs across contributor devices as a pull-based GitOps fleet. Each host runs the same container pair, polls a fleet repository, and converges on one origin; no host holds a privileged position on disk. This page owns the fleet model — the roster, the converge loop, digest pinning, the health worker, the network switch, the fleet mesh, the budget doctrine, automatic failover, and where lore is allowed to rest. Architecture owns the daemon inside each host; this page owns how many of them stay in step.

The roster: one line per host¶

The fleet is a file. fleet/hosts.yaml in a fleet repository (Infra.Dev today, Infra.Prod later) carries one line per host:

torneg: { owner: anward, role: primary, takeover: manual, age: age1ga25gx…kuk, key: "ssh-ed25519 AAAAC3Nza…GKWv nerthus-torneg" }
werbin: { owner: anward, role: replica, takeover: auto,   age: age1c68hx…tf7, key: "ssh-ed25519 AAAAC3Nza…Dz5u nerthus-werbin" }

• owner is a lore-repo member; provision re-checks membership on every run.
• role is primary or replica, and exactly one line is primary. Validation fails a roster with zero or two.
• age is the device's age public key; provision encrypts that host's tunnel token to it.
• key is the device's lore deploy public key; the primary's is the one write-enabled key.
• takeover is auto or manual (absent means manual) — the automatic-failover opt-in, covered below.

The primary is simply the first bootstrapped host. No host pre-exists the roster; the fleet is greenfield. A change to the roster is a merge request, and the merge is the approval — Bootstrap a fleet host walks the flow.

Town hostnames are first-level (torneg.nerthus.pl, and dev-torneg.nerthus.pl on the dev fleet) because free Universal SSL covers the root and one label only; a two-label name would force a paid certificate.

Everything the hosts share rather than hold per line — the apex, the zone, the lore repository, the hostname pattern, every tunable this page names — lives in fleet.settings.yml beside the roster. fleet.settings.yml is the key reference for that file.

The converge loop¶

Every host runs a small POSIX-sh agent, nerthus-agent.sh, on a five-minute timer (OnCalendar=*:0/5 with a 30-second jitter under systemd, the equivalent launchd job on macOS). Each tick:

1. Fetches the fleet repository's default branch and hard-resets its checkout.
2. Refuses, with a log line and a clean exit, if the contract's major VERSION outruns the agent.
3. Reads its rendered descriptor for the desired image digest, role, and sync mode.
4. Honors the upgrade wave: a primary host waits primary_lag_minutes (default 15) after the versions.lock commit before acting, so replicas soak a new image first.
5. Decrypts its tunnel token from secrets/<town>.tunnel.age with its device age key, in memory only.
6. Runs docker compose up -d, which no-ops when the pinned digest has not moved.

The device holds no cloudflared configuration file and no plaintext token at rest. The tunnel config lives at Cloudflare, written by CI; the token travels only as age ciphertext in git.

How many threads answer reads¶

reader_threads in fleet.settings.yml says how many threads a host gives the reads the route table marks poolable — per role, like sync_interval_min, and riding the same channel (settings → rendered descriptor → compose .env → the entrypoint's .nerthus/local/server.json → boot). Absent means the daemon's own default of none, which is every read answered on the accept loop, where one slow read makes every other caller wait for it.

Two, not more. Each reader is a full runspace inside the container's 3072 MB limit, so the ceiling is memory rather than the CPU share, and what the pool buys is concurrency — a probe not queueing behind a corpus read — rather than throughput. fleet:validate bounds the value to 0–4 and refuses a role name that does not exist, because a typo would otherwise render nothing and be discovered as a host that never started serving concurrently. Raise a replica before the primary: it publishes nothing, carries no scheduled work, and a wrong answer there costs no writes.

The effective numbers are on every host's anonymous /health as readerThreads and readersLive. They disagree exactly when a reader has died, which is the only symptom: the loop keeps serving everything itself, correctly and silently.

Derived work, copied rather than repeated¶

Some answers are expensive to derive and cheap to copy. The mention postings — word and stem to the transcript lines carrying them, behind GET /entities/{name}/mentions — are minutes of reading every transcript to build, a file to hand over, and identical on any host whose archive is identical. They are written only by a process that built the speech index, and every host after a restart adopts its index from cache instead, so before this nothing rebuilt them: a fleet that fetched transcripts settled into a corpus scan per mention read, measured on the dev hosts at 16 seconds, every read.

Each host now notices for itself. server.mention_index_interval_min (15 by default, 0 disables) is how often it checks that its postings still describe its own archive, and /health reports the answer as mentionIndex: memory, fresh, stale or absent. When they do not, the host tries two things in order:

1. Copy from a peer. GET /derived/{kind} serves the file a host already built for its own reads, under its own capability derive.read, carried by the fleet peer token the mesh already mints. Only a fresh artifact is served — a copy describing another archive answers 409 DerivedNotFresh rather than travelling — and the archive stamp rides a response header so a caller can check before reading megabytes.
2. Derive it locally, on the existing background job, if no peer has one. The fleet is an optimization here and never a dependency: a missing peer, an absent token, an empty roster and a refused payload all fall through to this.

The primary pulls; it is never pushed to. Replica deploy keys are read-only by design and replicas boot converge-only, and a standing credential that let any contributor laptop write into the publishing host would be a far worse thing than the latency it saves.

Importing an artifact is a performance decision rather than a trust one, and only because the consumer proves that is what it got: the kind must be on a closed list naming exactly one file it may write, the payload must parse, and its own internal stamp must equal this host's archive stamp — ordinals point into one concatenation of records and mean nothing against another, so a foreign payload would answer confidently wrong rather than slowly. It is written to a temporary name and moved into place, so a read in flight never meets half a document. Nothing else is writable: no lore, no state, no ledger, no tokens, and an unknown kind is a 404 rather than a path.

Digest pinning and the upgrade wave¶

The fleet pins the daemon image by digest, never by a moving tag — "a moved tag is a deploy nobody reviewed". versions.lock in each consumer repo holds it:

image: registry.gitlab.com/margonem/nerthus/it/nerthus.core@sha256:<64 hex — elided, see below>
schema_version: "0.11.0"
upgrade_wave:
  - replica
  - primary
# Built from Nerthus.Core main @ <sha>.

schema_version is the lore layout version the image expects, checked against each host's reported schema. A host whose on-disk index format outruns the pin goes read-only rather than write against a format it does not understand.

The digest is elided and schema_version is not, and the split is deliberate. A digest is a value nobody may copy — pasting one from a page pins a fleet to whatever image that page was written beside — so it is shown as its shape. schema_version is the opposite: the paragraph above says what a wrong one costs, so an example carrying a stale value teaches the exact mistake it warns against. It therefore carries the shipped value, read from Nerthus.Core.Infra.Dev versions.lock on 2026-08-28, and ci/assert-tracked-literals.py in the documentation repository fails the build when this literal and that file disagree.

New images arrive by proposal, not by push. A green Core build opens a merge request against the consumer repo that rewrites only the image: digest line and its provenance comment; the fleet's validate and plan jobs run on that MR exactly as on a human one. The dev fleet may auto-merge the proposal. The production fleet never does — its digest bumps are reviewed by a human. Set up pipelines owns the image-pin variables.

Releasing the contract¶

The scripts, the agent, the compose stack and the Workers live in the fleet contract; a consumer repository holds its roster, its settings and its pins, and no code. Cutting a tag on the contract is the release act, and a green pipeline there is what makes a tag safe to cut. Nothing reaches a fleet until that fleet's pin moves — a command under common/ci/ runs the pinned copy, not the contract's default branch.

A consumer names the contract in two places and no more: the ref: on the include:, and the common/ gitlink.

include:
  - project: 'margonem/nerthus/it/nerthus.infra'
    ref: 'v0.9.6'
    file: 'ci/fleet.gitlab-ci.yml'

include: ref: resolves before checkout, so it names the project and the tag directly; a path inside the common/ submodule cannot be include:local'd, because at include time the submodule does not exist yet. The submodule, checked out by the included template, is what the jobs execute. One bump is therefore two edits in one merge request: include: ref:, and the submodule to that tag's commit.

A fleet host runs the contract in its common/ submodule, and the version declared in CI is not evidence of the version running. The declaration is a fact about a pipeline. A host converges by fetching the fleet repository and updating that submodule on its own timer, so the two agree only while the host is still converging. What it executes is published anonymously as the contract block on GET /health — version, the submodule sha, and the agent hash — rendered by the converge agent out of the checkout it just converged. To answer "which contract is this host on", read the host.

contract:pin asserts that the two declared values agree, and it lives in the consumer's own .gitlab-ci.yml rather than under common/. It runs in .pre with no allow_failure, reads the ref: literal out of that file, reads the commit the runner checked out from common/.git, and resolves the tag through the contract project's public tags API. Every arm that has not compared two forty-character commit ids exits 1. Placing it outside the submodule is what makes it work: a guard that ships inside the artifact it polices is checked by whichever copy of itself a stale pin supplies.

fleet:validate carries a common-pin rule over the same comparison, and it is advisory. Three of its arms answer warning where the comparison cannot be made: no job token, an unreachable repository tree, an unreachable tags API. A rule that reports the absence of a check in the same green a passing check produces is not the thing holding the pin. Its prerequisite is the consumer project on the contract's CI/CD job-token allowlist (Settings → CI/CD → Job token permissions on the contract project). contract:pin needs no allowlist entry, because it sends no token: the contract project is public and an anonymous read of its tags is the whole call.

Both of those guard the pipeline, and a host does not consult one. It fetches this repository every five minutes on its own timer, so a gitlink committed backwards — which a working tree staged with a stale common/ produces without anyone deciding to — arrives whether the pipeline was red or green. The converge agent holds that half itself. Before it checks the submodule out, it reads the VERSION at the gitlink it is about to adopt and compares it against the release this host last converged, recorded per town at <state dir>/.contract.converged. A move forward is logged and taken. A move backwards is refused: the submodule is left standing where it is, the tick converges nothing, and because the checkout keeps the agent that made the comparison, every later tick reaches the same refusal. The host goes on serving the contract it already runs, and GET /health goes on publishing that version while CI declares the other one.

Tag names cannot carry that comparison. Twelve tags in the contract project declare a VERSION below their own tag name — v0.3.1, nine v0.6.x, v0.7.5 and v0.7.6, twelve of the thirty-six v* tags that exist at v0.9.7 — so a floor built on the tag would compare two different quantities on a third of the releases. VERSION against previously-converged VERSION compares one, and it needs no API.

A rollback ships as a version above the one it undoes. There is no per-host override and no flag to pass: with the record removed the comparison falls back to the VERSION the checkout is standing on, which is the same answer. Cut the replacement release above the current one and the fleet takes it as an ordinary forward move.

The health worker¶

Continuous health lives in a Cloudflare Worker on a fifteen-minute cron (*/15 * * * *, 96 sweeps a day), not in CI. It ran at one minute until 2026-08-20; Check the quota holds why it moved and what the move cost. The worker reads the roster's expectations from KV (written by provision), then calls GET /v1/api/health on each host — an anonymous endpoint, so the worker holds no per-host token. It compares reachability, sync freshness, image digest, and schema against expectations.

The KV write is decoupled from the cron: the worker writes the state document on a real state change and, unchanged, once per 15-minute heartbeat floor — about 96 writes a day against a free cap of 1,000 (Check the quota owns the budget arithmetic, and holds the week in August 2026 when a clock left in the change comparison made that decoupling untrue and blew the cap seven days running). The worker posts to Discord on transitions only plus one daily summary line. The summary doubles as the sweep's dead-man heartbeat: a missing summary means the worker itself is broken. When a host's observed digest first matches a pending provision, the worker posts a convergence tick and clears it. A pin still unmet after the fleet's own convergence budget — the slowest tick plus primary_lag_minutes, times three, and never under 45 minutes — draws one warning; the factor is what leaves room for a laptop that slept through a tick or two. The warning splits by what the sweep can tell apart: a host that answers and serves the wrong digest has an applier that is refusing and its own log says why, while a host that does not answer is dark and nothing about its image is known.

The standby gate. The worker is also built to stand down. Before sweeping, its cron probes the self-hosted monitor's published state document on the monitor host's own tunnel hostname — never the apex, which is the very thing failover moves, and never KV. Fresh state (a sweptUtc under 40 minutes old — two missed 15-minute monitor sweeps, hysteresis against single-sweep jitter) means the worker stands down at zero KV operations; stale, unreachable, unparsable state — or the probe hostname simply unset — means it sweeps exactly as described above. Fail-active is the rule. The monitor box exists and sweeps as of 2026-08-20, and the gate is still dormant on purpose: both watch, the two are compared hourly, and arming waits on a week of agreement — because the worker is the fallback and a fallback that disagrees with the primary is worse than none (the cutover runbook's territory; The two watchers is the model). In standby the worker becomes the fleet's dead-man watcher: the moment the monitor goes dark, the very next sweep takes over — the standby↔active edges each post one transition notice.

Reading the fleet in one place¶

The worker sweeps and alerts; it does not answer questions. What the roster asked for, what provision recorded, what the sweep last saw and what each host says about itself live in six places, and the finding is almost always a disagreement between two of them rather than any single value. common/ci/Get-NerthusFleetState.ps1 reads all of them and prints the comparison.

Secretless by default, on the same trade fleet:plan makes: the committed documents, each host's anonymous /health and /fleet/status, and the worker's public GET — whose URL is health_worker_url in fleet.settings.yml, or NERTHUS_FLEET_HEALTH_URL, or -HealthWorkerUrl. -Token (the it bundle, defaulting from NERTHUS_IT_TOKEN) adds /sync and /logs/status — the commit each host serves, its clone against origin, and its transcript-fetch health. Every host verifies its own minted tokens, so a fleet-wide read takes the map form torneg=<token>,werbin=<token>; a bare token is used for every host, which is right for a fleet-wide credential and wrong for a machine one. A tier whose credential is absent drops its columns and says so as a warning, the way the submodule-pin assert does: a reader must still answer what it can see.

-Town reads one host instead of the fleet, -Offline reads the committed documents alone, and -Format json is for anything that is not a terminal. Exit 0 means every host agrees, 1 that there is at least one finding, and 2 that a source could not be read at all.

Two things it reports that nothing else did. The commit, because sync age says a tick succeeded and never what it converged onto — two hosts can both read fresh on different commits. And descriptor drift: fleet:validate asserts a committed descriptor matches current settings, while nothing asserted that the host is running what its descriptor says, and a host takes its image and its compose environment separately.

One more tier, behind its own credential. -GitLab (a read_api PAT from NERTHUS_GL_READ_TOKEN, or CI_JOB_TOKEN inside a pipeline) reads what only the control plane knows: whether the last fleet:provision succeeded, whether a fleet:rotate followed it — job order is the only place that fact lives — the open merge requests that move the pin or the roster, the lore project's deploy keys, and this month's shared-runner minutes. It opens nothing and merges nothing, and it never uses the Worker's GL_FLEET_TOKEN: that credential exists for one rare actuation path and a monitor must not widen what it can reach.

-Follow tails the fleet's journals instead of printing state: one NDJSON line per event, merged across hosts, a cursor per town so a run resumes where the last stopped, and a journal:gap line where a cursor fell behind the oldest entry still in reach. It needs -Token, because the journal costs events.subscribe. It polls every -IntervalSec seconds (15 by default) until interrupted, or once and returns under -Once. A poll that reaches no journal says so on stderr rather than printing nothing, since an empty stream and a quiet fleet look identical.

Its verdicts are one vocabulary — dark, skew-image, skew-schema, vantage-split, lore-divergent, diverged, dirty, fetch-stale, stale, corpus-divergent, descriptor-drift, degraded, restarted, absent, orphan, role-drift — ranked by severity, so a row leads with the fact that explains the others. A dark host earns dark and nothing else: everything else about it is a memory of the last time anyone could look. vantage-split is a host the three vantages disagree about — peers from inside the fleet, the fleet-health Worker from the edge, the restoration monitor from an always-on box — and it ranks with the drift states, below dark and above stale.

The event journal¶

Each host appends the durable events it publishes to .nerthus/log/events.jsonl and serves them by sequence from GET /events/journal?since= (API reference). The SSE stream keeps a bounded replay window, which is enough for a consumer that is connected and nothing at all for one that was away — and a fleet of laptops is mostly hosts that were away. A reader keeps one cursor per host, and a cursor older than the oldest line still in reach is told so (truncated) rather than answered with silence that reads as quiet.

A sequence counts what one host published, so the route is deliberately off the replica flag: an edge free to pick a host would answer one host's cursor with another's events.

The network switch¶

Whenever the live daemon behind evocation.nerthus.pl can do the work, it does; only when it cannot does the Worker path serve — with no operator action in either direction. The Workers stay deployed permanently as the fallback layer, so no capability is lost while daemon-side serving becomes the norm. The switch takes a different shape per class, because the classes fail differently:

• Discord. Discord allows exactly one interactions endpoint, so the report Worker is the router: it keeps its signature verification and its 3-second ack unchanged, then offers the interaction to the live daemon on a 6.5-second attempt. Acceptance means the daemon enacts in-process and posts the followup itself; anything else — a timeout, a refusal, a read-only or stale daemon — falls through byte-identically. What a decline costs depends on the kind. A close and a report fall back to the existing pipeline trigger, so a decline costs one CI minute and nothing else. A query (/status, /sesje-otwarte, /pu) has no fallback by construction — a whole ephemeral-daemon pipeline per question does not fit the budget — so a dark fleet answers that it is unavailable. The followup's tail tells you which path enacted: a daemon enactment ends with ⚡; a CI one is unmarked, exactly as before. The daemon refuses dispatches older than 5 seconds (receipt-relative — clock skew between the edge and a sleeping laptop must never silently eat work; skew beyond 2 s is logged loudly instead), and refuses a report older than 3 seconds on top of that: a close is idempotent at the far end, so a race with the Worker's timeout is free, while two report enactments would be two merge requests for the Rada to untangle. A report the daemon does publish is a zgloszenia/<stamp> merge request it pushes itself, cut from freshly converged main and followed by a reset back to origin — which is also why it refuses outright when the host's worktree holds anything unpublished.
• CORS / dashboard. No switch exists to build — the CORS shim fronts the daemon on the same hostname, and when the daemon is down there is no alternative origin. The class gets idempotent layering instead: the edge grants CORS headers only when the upstream response does not already carry them, and stamps x-nerthus-cors: edge when it granted (absence means the origin did). A future daemon-side CORS middleware can land under the worker without double-header breakage.
• Fleet health. The standby gate above: the worker sweeps only while the monitor is dark, and stands down at zero KV cost otherwise.

The daily fleet-health summary carries the measured share: each daemon publishes per-kind dispatch counters on its anonymous /health, and the sweep sums the enacted ones into one dyspozycje Discord: N ⚡ line.

What did not change: Ed25519 verification stays at the edge, the trigger pipelines remain fully functional and unexercised (the monthly canary this sentence used to describe was never built — see Read the fleet alerts), and a fully dark fleet still processes /zgłoś through the Worker→CI path exactly as before the switch existed.

The fleet mesh¶

Hosts have eyes on each other. Each daemon serves an anonymous per-host status card — role, time, uptime, boot id, image digest, schema, sync age; deliberately nothing more, and in particular no takeover-MR coordinates on any anonymous surface — and runs a peer watch: a periodic probe of one peer per tick, round-robin, over the public per-town hostnames, recording reachability, measured clock skew, and role claims. The aggregated observations (a who-is-dark-right-now map is reconnaissance, so it costs a capability) serve from cache only — a peer route never proxies a live request, which keeps the single serve thread out of hostage situations and the public hostname out of the SSRF business.

The cadence is fleet configuration, not code: peer_watch_interval_min in fleet.settings.yml, riding the exact channel sync_interval_min already rides (settings → rendered descriptor → compose .env → daemon env). The interval is the gap between one host's successive probes; each probe hits one peer round-robin, so a given peer is re-observed every interval × (N−1) minutes — at the default 5 with 3 hosts, every ~10 minutes. An absent key means 5 everywhere with no error; a present-but-out-of-range value (the bounds are integers 1–60) fails fleet:validate before it ever renders; a bad value that somehow reaches a daemon's environment is clamped back to 5 with one boot log line naming it — never a crash, never silence. The effective value is observable in the boot log and on the peers aggregate. The probe timeout is 2 seconds and deliberately does not scale with the interval — it bounds how long a probe may occupy the serve thread, which is a per-tick property, not a cadence one. Peer-dark alerting fires on the edge of 3 consecutive misses (so the alert lag is 3 × interval × (N−1), and tightening the interval tightens detection with no code change). Measured skew is a naive half-RTT estimate — good enough to flag clocks more than 2 seconds apart, worthless below ~200 ms over a proxied tunnel — and it is observability only, never an input to any enforcement decision.

Writes go through git, hints go through HTTP. POST /fleet/takeover makes a daemon propose the apex flip through exactly the git-anchored flow the health worker uses — same candidate ranking (the worker's implementation is canonical; a shared fixture set keeps the daemon's port in verdict-for-verdict agreement), same open-MR pre-flight, same cooldown, same auto-merge window — with this host's own corroborating peer observations embedded in the MR description so the human in the window sees why. An unforced request requires that corroboration (the primary observed dark for the same 3-miss streak, or the primary itself stepping down); force skips corroboration only — never the pre-flight, never the cooldown — and exists for the split-brain repair case where corroboration is exactly what is broken. POST /fleet/announce is the write side's echo: the proposer tells the other hosts "topology changed", and each reacts by verifying, not believing — an announcement triggers an early sync tick, never a role change. Git remains the only authority on roles; a forged or stale announcement costs one early sync and a log line. Split-brain — two fresh cards claiming primary — alerts and records, never acts, for the same reason.

The budget doctrine¶

The fleet is built to the GitLab Free tier's 400 CI-minutes per month. That budget closes only by keeping expensive work out of CI:

• Health and deferred verify run in the Worker cron. A CI-based health sweep alone would cost several times the whole monthly quota.
• Map checkup and close reminder run as the primary's in-daemon ticks, which is what schedules.map_checkup.owner and schedules.close_reminder.owner say in committed config. On shared runners the CI form would cost multiples of the quota; a self-hosted runner takes that cost off it.
• Settlement stays a monthly CI schedule (17 3 1 * *); it is cheap and runs once.
• The lineage autopilot runs beside it (schedules.lineage_autopilot.owner, default fleet): memberships and origins the format-11 events prove on two sessions and two logs land as (auto) lines on the dev lore only, and the batch they ride in is asked to merge on green (Entities).
• The alias autopilot runs as an in-daemon tick on every host that publishes (schedules.alias_autopilot.owner, default fleet): exact, rival-less computed aliases land as (auto) lines, at most twenty per half hour (the alias autopilot).

A fleet merge costs about three minutes. Check the quota is the monthly reading procedure; Set up pipelines owns the schedules themselves. When the budget still runs short, a self-hosted runner is the structural escape valve — its minutes do not count against the 400 — but it is capacity, not a fleet host, so it lives outside this model in Set up a self-hosted runner.

Automatic failover¶

When the primary goes dark, the health worker can promote the best replica without a human awake. Git stays the only authority: the worker proposes a role flip as a merge request, and the merged pipeline's provision does the switch.

Detection. The bound is wall-clock: the worker arms when the primary has been dark for auto_failover.after_minutes (default 30), anchored on the last sweep that saw the primary up, with no takeover in flight or cooling down. A settings file still carrying the legacy auto_failover.after_sweeps is converted at 15 minutes per sweep — the cadence that key was written against — whatever the current cron, so the dev fleet's after_sweeps: 2 means 30 minutes. A single missed sweep is a sleeping laptop, not an outage. The failover thresholds are rendered into every host descriptor as well as merged into the worker's KV config, so the daemon's own corroboration gate and the worker reach the same verdict about when a primary is dark — a threshold honoured by one proposer and not the other is two proposers disagreeing.

Re-arming. Arming can fail for reasons outside the worker: no replica qualifies (blocked), or actuation itself fails on a bad token or a GitLab 5xx (failed). Both are waiting states, not stops. The worker returns to idle once the primary answers again or the cooldown that each of them starts has elapsed, so a transient flake costs one cooldown rather than switching failover off until a human edits hosts.yaml.

Eligibility. A replica is a candidate only if it opts in with takeover: auto and passes every hard filter: reachable, image matches the pin, schema matches, and sync is fresh. takeover: auto is the Rada's judgment that a device is steady enough to write from — a battery-bound laptop stays manual.

Selection. Among eligible replicas the first differentiator wins: the 30-day availability ratio (used only once some host has 7 days of coverage), then sync freshness, then longest continuous uptime, then roster order as a deterministic final tiebreak. On a young fleet with no host yet at 7 days, selection falls back to longest uptime.

The availability ratio is the health worker's key, computed from a 30-day day-bucket ledger it keeps in KV. The daemon-side proposer behind POST /fleet/takeover keeps no such ledger, so its ranking is uptime first, then sync freshness, then roster order. The two agree on eligibility, which is the filter that decides whether a host may be promoted at all; they can differ on which of several eligible replicas they nominate. Whichever proposes first is adopted by the other through the pre-flight below, so the difference costs a choice between qualified hosts, never a two-primary roster.

Promotion. The worker opens the flip as branch takeover/<town>-<stamp> with MR title Zmiana hosta APEX: <primary> → <town>, auto-merges it when the pipeline is green, and provision re-points the apex CNAME and re-renders both roles. End to end the window is 40–55 minutes. A cooldown_hours of 24 blocks a second auto-promotion, so a flapping primary cannot ping-pong the apex, and an epoch counter gives the Rada a lineage.

The pre-flight. Before creating anything, every proposer lists the standing takeover/* merge requests: an opened one is adopted (its number recorded, no second branch), and one merged within the cooldown is honoured as the takeover it was. This listing is the only guard against two proposers merging a two-primary roster — validation's exactly-one-primary rule runs on each branch's own content, and two individually-consistent flips from the same baseline both pass it — so the pre-flight ships with the fixture proving that defect, and an adoption posts its own notice: the guard held, but two proposers overlapped, and an operator should know.

Manual stays the fast path. When a human is present, a hand-edited role-flip MR is faster than waiting out the detection window. Disabling automation mid-incident is itself a one-line MR: set auto_failover.enabled: false in fleet.settings.yml, and the same outage produces alerts only. Run a failover is the procedure for both paths.

The tech notification channel¶

Fleet events land in a dedicated Discord channel, kanal-wewnetrzny-it, owned by the IT contributor role — a read-only role that observes the fleet and delivers notices without authoring lore. Infra noise stays off the Rada's channels. Both producers, the CI notify job and the health worker, resolve the channel by name in the guild and post as the bot (DISCORD_BOT_TOKEN plus DISCORD_GUILD_ID); there is no webhook to keep in sync. Each notice links the pipeline or MR that produced it. Manage notifications holds the wiring and the full event catalog.

Failure rows¶

Each row is one situation and its procedure; the runbook owns the steps.

Data residency¶

git log fleet/hosts.yaml is the residency register. Lore at rest exists in exactly two kinds of place: GitLab, and the Docker volumes of the devices hosts.yaml enumerates — devices whose owners are membership-verified and whose lines the Rada merged. Nothing else holds a copy. The Infra pipelines and the health worker never clone lore; they touch the roster, Cloudflare, and the anonymous health endpoint only. Cloudflare's cache-bypass rule means the edge stores nothing. To answer "where is the lore?", read the roster's history.

A fleet whose settings carry mc_repo_url puts a second body of data on the same hosts: the moderation corpus — the /mc clone holding the moderator roster, the versioned regulations, and the sanction record. Its deploy keys are read-only on every host, and jadzia.nerthus is the only account that writes it. fleet.settings.yml owns the key.

How many corpora a host holds is that fleet's decision, so the answer is a measurement rather than a capability. The dev fleet on 2026-08-23 — torneg, karka-han and eder, on Infra.Common v0.9.6 — mounts six named volumes per host and serves four trees. Every count below is tracked files in that checkout, read with git ls-files at the commit named beside it, and every one is identical on all three hosts:

Read the archive's 2 330 with care, because three populations live in it and they are not multiples of each other. 1 163 fetched transcripts sit under nerthus.logs/, each as a raw .log beside its parsed .json sidecar — 2 326 files. Beside them is one sidecar with no transcript, i.imgur.com_f6cpnvF.png_….failed.json, a PermanentlyFailed marker for a PNG somebody pasted as a transcript, which is what makes nerthus.logs/ 2 327 rather than 2 326. One further .log is hand-saved under Logi/ and has no sidecar, so a tree-wide extension count reads 1 164 of each. That pair looks like 1 164 paired transcripts and is not — the extra .log and the extra .json are different files, in different directories. README.md and .gitattributes are the last two.

/dziel and /adnotacje are mounted and hold nothing: dziel_repo_url and adnot_repo_url are unset, so the entrypoint writes no corpora.json key for either and each tick returns before it builds a sync view. Empty there is the state the fleet chose, not a clone that failed; Infra.Dev's own fleet.settings.yml carries the argument for it.

Each host's cloudflared keepalive costs roughly 20–30 MB/day, worth noting for a host on a metered or LTE connection.

Shelved designs¶

Three fleet designs are recorded and not built; each names the trigger that would revive it.

• A Durable Object lease with epoch fencing would give writer failover a hard fence instead of the bounded dual-primary window the MR-mediated flip accepts. It stays shelved as the escalation if the 40–55 minute promotion proves too slow in practice.
• SSE fast-converge would push a replica's staleness down to seconds. It is gated on a ten-minute named-tunnel test of the existing /events endpoint; a known cloudflared bug buffers server-sent events over quick tunnels, and named-tunnel behavior is unverified. If it fails, the fallback is a 30–60 second poll of a cheap endpoint that arms a converge on change. Interval polling stays underneath either way — the stream is a latency hint, never the correctness mechanism.
• Queues map-sweep sharding would fan the map checkup out across name-sorted slices. It ships only if the primary's in-daemon sweep proves inadequate after the cadence fix and the burn-in.

See also¶

• Architecture — the daemon each host runs
• fleet.settings.yml — every key beside the roster, and what each absence costs
• Sync — how a host converges with origin, and the freshness gates
• Bootstrap a fleet host · Fleet runbooks
• Publish the API — the edge the fleet sits behind