Skip to content

Rotate a root credential

A root credential is one that mints others and that no rotation of theirs replaces: an account login, its two-factor recovery codes, a provider login, the recovery age identity. They stay out of the vault entirely (ruling 39) — out of sops, off every host, out of CI — and in a person's keychain or on offline media. Every one of them has a tier: root row in Nerthus.Infra/vault/inventory.yaml naming its holder, its ceiling and the section of this page that rotates it.

nerthus secrets verify refuses a root credential whose value is in a vault. The absence of one is the correct state and is reported as unprobeable with the reason, not as a missing value — the two look alike in a report and mean opposite things.

What is root, and what is not

Credential Issuer Ceiling Holder
GITLAB_STEFAN_NERTHUS_LOGIN gitlab 180 days operator's keychain
GITLAB_STEFAN_NERTHUS_2FA_RECOVERY gitlab 365 days operator's keychain
CLOUDFLARE_ACCOUNT_LOGIN cloudflare 180 days operator's keychain
DISCORD_APPLICATION_OWNER_LOGIN discord 180 days operator's keychain
LUCKYSRV_ACCOUNT_LOGIN luckysrv 365 days operator's keychain
INSTANTNODE_ACCOUNT_LOGIN instantnode 365 days operator's keychain
NETCUP_ACCOUNT_LOGIN netcup 365 days operator's keychain
MIKRUS_ACCOUNT_LOGIN mikrus 365 days operator's keychain
NERTHUS_RECOVERY_IDENTITY none — generated 730 days offline media

A scoped API token is not root, and the distinction is deliberate. nerthus-n (GitLab, api + create_runner) and the account-owned Cloudflare token both expire, both are scoped, and both are rotated by minting a replacement at the account above — so the thing that cannot be replaced is the account, and that is where tier: root sits. On GitLab Free an api token can still mint deploy keys, deploy tokens and runners; that is a real widening and it is watched as an open line rather than hidden by relabelling the token.

Before any rotation

  1. Read the row: nerthus secrets verify --inventory Nerthus.Infra/vault/inventory.yaml prints the population, the ceilings and which rows are root.
  2. Know what breaks between old and new. Every rotation below has a window where the old value is gone and the new one is not yet everywhere; the sections say how long.
  3. Never paste a value into a terminal a session can read, into a chat, or into a file this repository tracks. A value that passes through any of those is rotated again, immediately — this estate has done exactly that.

GitLab: the machine user's login and recovery codes

The account mints every GitLab credential the estate holds.

  1. Sign in as stefan.nerthus and change the password in Edit profile → Password.
  2. Regenerate the two-factor recovery codes in Account → Two-factor authentication; the old codes stop working the moment the new set is shown.
  3. Store both in the keychain entry the inventory row names, replacing the old entries rather than adding beside them.
  4. Prove it: sign out, sign in with the new password and one new recovery code, then discard that code — a spent code is spent, and using one is the only proof there is.
  5. Nothing else changes. Existing PATs, deploy keys and deploy tokens survive a password change; this rotation costs no downtime.

Cloudflare: the account owner login

  1. Sign in and change the password; regenerate the two-factor recovery codes with it.
  2. Prove it by signing in again.
  3. Existing API tokens survive. If the account's email changes, every token stays valid and the inventory's accounts: row is what has to move.

Discord: the application owner

  1. Change the password on the owning account, then regenerate its recovery codes.
  2. The bot token is not rotated by this. Re-minting one is a separate act in the developer portal, and it takes the bot offline until every holder has the new value — nerthus discord register and the site's own store are those holders.

A provider login

  1. Change the password at the provider's panel and store it in the keychain entry the row names.
  2. Console access is what this credential is. It does not reach a host that answers a key, and a host whose sshd is broken is recovered by reinstall — host adopt rotates the initial root password and discards it, so the panel's password prompt opens nothing on an adopted machine.

The recovery age identity

This is the one that reads every value the estate holds. Rotating it rewrites every ciphertext in vault/.

  1. Mint the new identity on the machine that will hold it, and write the private half straight to offline media: age-keygen -o /Volumes/<media>/nerthus-recovery.key. Two keys on one disk is one key.
  2. Put the public half in Nerthus.Infra/inventory/vault.yaml as the recovery row, replacing the old one, and render the rules: nerthus-config vault render-rules . --write.
  3. Re-key every file, from inside vault/: cd vault && for f in $(git grep -l '^sops:' -- '*.yaml'); do sops updatekeys -y "$f"; done, with the OLD identity still available — sops has to decrypt before it can re-encrypt. Two traps, both measured: sops reads .sops.yaml from the working directory and not from the file, so the same loop run from the repository root answers «config file not found»; and a '*.sops.yaml' glob misses the Core-era files, which are *.yaml and are ciphertext all the same.
  4. Read the recipients back by fingerprint, from a fresh clone rather than from the tree that wrote them: nerthus-config vault check . reports the population it judged and every file encrypted to a key no row declares.
  5. Prove the new identity opens one real file, report the count and never the value, and only then destroy the predecessor's media.
  6. Between steps 3 and 5 the estate has two readers. That is the safe order: removing the old reader before the new one is proved is how a vault becomes unreadable.

When a holder leaves

Removal from a recipient list is not revocation. A person who could read a value can still read every copy they ever decrypted, and git history keeps the old ciphertext for ever. So:

  1. Rotate every root credential that person held, by the sections above.
  2. Re-mint every vault credential they could decrypt — nerthus secrets offboard <person> computes that set from the inventory and diffs the issuers against it.
  3. Remove their key from inventory/vault.yaml, render the rules, re-key the tree as above.
  4. Record what was rotated and what could not be, in the sprint ledger. A rotation makes a leaked value worthless; it never makes the history clean, and saying so is part of the procedure.

The ceiling reminder

Each row carries ceiling_days. The monitor reads the ceilings — never the values — and posts to #szary-monitoring when a root credential is within thirty days of its ceiling. A ceiling is a plan and not a verdict: only the act of using a credential says whether it still lives, and for a root credential that act is a person signing in.