Rotate secrets¶

Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: not yet written.

Rotate each fleet secret on its own clock, by its one correct path. The Discord bot token is the one held in several places at once, so this page is most careful about it. Fleet owns the fleet model these secrets serve.

A host's tunnel token¶

Rotate when a device is lost or wiped, when the token may have leaked, on schedule, and after a provision run that rewrote a host's ciphertext.

1. Run the manual fleet:rotate job on the fleet repo's default branch with ROTATE_HOST set to the town. all is the other accepted value: every town the ledger names, in one job and one commit-back. Reach for it when the encrypting side changed and every committed token went unreadable at once, because town by town means one manual job each and a half-rotated fleet in between.
2. The job re-mints the tunnel secret at Cloudflare, re-encrypts the new token to that host's age: key, and commits the ciphertext back.
3. The device converges within five minutes and cloudflared restarts on the new token.

Expect up to one tick of downtime for that host: the old token stops working the moment the secret is re-minted.

Provision rewrites a ciphertext only when it has to. A host's tunnel token is re-encrypted when its file is missing, when it was encrypted to a recipient absent from the current roster, or when it describes a tunnel absent from the current ledger; a zero-change provision touches no secret, so a ciphertext nobody rewrote cannot have been rewritten wrongly. Before any write, provision proves its own encrypt path with a throwaway-key age round trip. What that round trip cannot prove is the devices' side, which is why a provision that did rewrite a ciphertext still wants the rotation after it: only a device decrypting the committed file settles that it can.

Get-NerthusFleetState.ps1 -GitLab reports a provision with no rotation after it as ciphertext-suspect, a warning rather than an error (Recover a diverged host owns reading that report) — job history cannot see whether the run rewrote anything, so the warning is conservative and a no-op provision draws it too. The check reads git as well as job history, so a rotation committed from a workstation clears it exactly as the job does. One commit never counts: chore: fleet provision state is the provision's own commit-back.

A device's age key¶

For a wiped or lost device. It cannot decrypt anything encrypted to a key it no longer holds, so this is a re-join, not a repair.

1. On the device, re-run sudo /opt/nerthus/infra/common/bootstrap/bootstrap.sh <town>. It generates fresh keys and prints a fresh roster line.
2. Open a merge request updating that host's age: and key: values in fleet/hosts.yaml.
3. On merge, provision re-encrypts the tunnel token to the new age key. Rada replaces the old deploy key on the lore repository with the new one.
4. If the device was lost rather than wiped, also rotate the tunnel token above: the old device could still hold a decryptable copy.

Revocation is one deletion. An age key no longer in hosts.yaml receives no future ciphertext, and a deploy key removed from the lore repository opens nothing.

The Discord bot token¶

The fleet's status notices post to the kanal-wewnetrzny-it tech channel as a bot POST resolved by name, with no webhook (Manage notifications owns the wiring). One bot token drives every sender, so it is held in three places:

• A wrangler secret on the health Worker: DISCORD_BOT_TOKEN.
• CI variables on each fleet repo: DISCORD_BOT_TOKEN (masked) and DISCORD_GUILD_ID, read by fleet:notify.
• The fleet secrets channel for the daemon: secrets/<town>.discord.age in the fleet repository, encrypted to the primary's device age key. The agent decrypts it and renders DISCORD_BOT_TOKEN into the compose .env on the primary only — a replica boots -NoDiscord and gets none — and an absent file renders empty rather than failing the converge.

It is the same identity everywhere. To rotate:

1. Reset the token in the Discord developer portal. The old token dies at once, so expect a brief gap in notices until every holder below is updated.
2. wrangler secret put DISCORD_BOT_TOKEN -c worker/wrangler.toml, pasting the new token.
3. Update the DISCORD_BOT_TOKEN CI variable on every fleet repo (dev and, later, prod).
4. Re-encrypt the new token to the primary's age key as secrets/<town>.discord.age in the fleet repository; the primary's next converge renders it into the compose .env.

5. Confirm delivery: run a fleet pipeline and watch its fleet:notify line, then wait one sweep for a Worker line. A sweep is fifteen minutes (*/15 * * * *, worker/wrangler.toml), so give it that long. Both prove the new token before you walk away.

   No Worker line after one minute means nothing at all. The sweep has almost certainly not run yet. Only a silent channel fifteen minutes after a fleet:notify line that itself succeeded is evidence of anything, and even then read it as the Worker has not swept or cannot post, not as the token is bad — the Worker discards its Discord response, so a 401 from a botched rotation resolves as a success on its side.

If you ever re-invite the bot, keep its permissions at View Channel + Send Messages (3072). A zero-permission invite leaves it in the guild yet unable to see any channel, so every resolve finds nothing and posts nothing.

The machine-user PAT¶

NERTHUS_INFRA_PUSH_TOKEN is a personal access token on the bot account (api + write_repository), because project access tokens are paid on gitlab.com. It expires within 365 days; put that expiry on a calendar.

1. Mint the replacement on the bot account before the old one expires.
2. Update the CI variable on every repo that holds it (the fleet repos; the lore repo's settle and close pipelines carry their own).
3. Run a fleet pipeline and confirm the provision commit-back lands.
4. Revoke the old token.

An expired push token looks like a provision that succeeds against Cloudflare and then fails to commit its state, which leaves the ledger behind reality; Recover a failed commit-back is the repair.

The health Worker's fleet token¶

GL_FLEET_TOKEN is a wrangler secret on the fleet-health Worker: a machine-user PAT (api scope, Maintainer on the consumer repo) the Worker spends only to open and auto-merge the role-flip merge request on automatic failover. It is the same bot identity as NERTHUS_INFRA_PUSH_TOKEN but a separate token instance, so revoking one leaves the other alone.

1. Mint a new PAT on the bot account before the old one expires (≤365 days, a calendar entry).
2. wrangler secret put GL_FLEET_TOKEN -c worker/wrangler.toml, pasting the token over stdin.
3. Revoke the old token.

There is nothing to confirm between steps 2 and 3: the Worker reads the secret on its next sweep, and the failover path is exercised only during an outage. While this token is absent or invalid the Worker still sweeps and alerts; only the automatic takeover is unavailable, and the manual role-flip stands in for it (Run a failover).

The daemon dispatch token¶

DAEMON_DISPATCH_TOKEN is a wrangler secret on the report Worker: a named daemon token (capability discord.dispatch, in no role bundle) the Worker presents when it offers a close interaction to the live daemon before falling back to the pipeline trigger.

Mint and set — one piped act, the value never echoed:

1. Against the live primary, POST /tokens with { "name": "worker-dispatch", "capabilities": ["discord.dispatch"] }, piping the response's raw token straight into wrangler secret put DAEMON_DISPATCH_TOKEN -c vm/report-worker.wrangler.toml.
2. Provision the record file to every primary-capable host through the fleet secrets channel (the same age-encrypted path that delivers the tunnel token). The token's record is just a file the daemon's resolution reads, so a host that has never minted anything still authenticates the Worker — which is what keeps dispatches working after a failover.

After a takeover. If the record was not provisioned to the new primary, every dispatch starts failing authentication and the system silently pins to the CI fallback. The repair is a re-mint against the new primary (step 1 again) plus the record provisioning (step 2). The takeover drill rehearses exactly this.

Rotation: re-run step 1 (a new mint is a new value; the old record file is revoked by deleting it via DELETE /tokens/{id} or removing it from the provisioned set).

The fleet mesh credentials¶

Two credentials each daemon reads from its own environment, both loud in their absence:

• The fleet peer token (NERTHUS_FLEET_PEER_TOKEN, every host): the one host-to-host credential. Each daemon derives a fleet-peer token record from it at boot, so the same value authenticates incoming peer calls and is the bearer on this host's outgoing announcements and derived-artifact pulls. The record carries fleet.read, fleet.announce and derive.read, never the takeover-request right, so a stolen peer credential observes and hints rather than requesting topology writes. Unset, the daemon logs no NERTHUS_FLEET_PEER_TOKEN — announce fan-out skipped and leans on the agent cadence.
• The daemon's GitLab actuation credential (NERTHUS_GL_FLEET_TOKEN plus NERTHUS_FLEET_PROJECT_PATH, primary-capable hosts): GL_FLEET_TOKEN's twin, a separate PAT instance on the same bot account, so the Worker's and the daemons' actuation credentials rotate and revoke independently. With either half missing the daemon's takeover route answers 503 rather than degrading quietly. NERTHUS_FLEET_API_BASE and NERTHUS_FLEET_DEFAULT_BRANCH are optional and default to https://gitlab.com/api/v4 and main.

No part of the provisioning path delivers either value, so there is nothing on a host to rotate. The agent renders the compose .env from the host's descriptor, and that block carries the tunnel token, the roster, the cadences and the failover thresholds. The compose stack passes neither the peer token nor the actuation PAT into the container.

Both are therefore absent on a stock host: no fleet-peer record is minted, POST /fleet/takeover refuses, and the health Worker's own GL_FLEET_TOKEN is the only actuation credential the fleet holds. Manual promotion is the path that still works (Run a failover). Delivering these two credentials means extending the age-encrypted secrets channel to carry them.

PAT expiry watch list¶

Every dormant credential rots on its own clock, and the fallback legs are exercised rarely enough that an expiry would otherwise be discovered during the outage the fallback exists for. Calendar all of these at mint time:

The Cloudflare API token¶

Scopes: Tunnel Edit, DNS Edit, Zone Ruleset Edit, Workers Routes Edit, Workers KV Edit, SSL Read. It lives only as a protected, masked CI variable on the fleet repos, never on any device. Mint the new one, update the variables, run a pipeline (provision is idempotent, so a no-op run proves the token works), then delete the old token.

SSL Read is the scope a hand-written token loses most easily, and it is read once, at the end of the run: provision closes by asking the zone whether Universal SSL is on, because every fleet name is first-level precisely so the free certificate covers it. That read is unguarded, so a token without the scope throws there — after the tunnel, the DNS record, the secrets and the edge rules have already been written. What you get is a half-applied provision, not a refused one, which is why the no-op run matters before you delete anything.

See also¶

• Fleet — the model these secrets serve
• Manage notifications — where the bot token is spent
• Offboard a host — the tunnel-token rotation a lost device forces
• Recover a failed commit-back — when a rotation re-minted the token and never published the ciphertext