Move a runner between machines¶

Relocate a runner to another host. Not a registration (Set up a self-hosted runner) and not a retirement: the runner keeps its GitLab id, its history and its authentication token, and only the machine polling with it changes.

A runner's id in the declaration is <host>:<name>, so moving one is a re-identification: the old row goes, a new row appears under the new host, and GitLab is told the new description. Doing half of that leaves a runner nothing answers to.

Before you start¶

Prove you can write the runner API. The cheap decisive test is a no-op PUT of the runner's current description:

glab api --method PUT "runners/<id>" -f "description=<its current description>"

If that fails, stop. You would be able to move the machine and unable to re-describe the runner, which is the combination that leaves jobs pending.

Have the token off the machine you are emptying. It is in vault/hosts/<old-host>/ (The secret tree). If its only copy is on the host you are about to wipe, you are one disk away from re-registering the runner and losing its history.

Drain in-flight pipelines. A job running when its supervisor stops fails.

1. Give every tag an online holder first¶

A tag naming no online runner is not an error. The job sits pending and fails out at the timeout as stuck_pending_no_matching_runners, which reads as a hung queue rather than as breakage. Park the tags on a runner that is staying up for the length of the window:

glab api --method PUT "runners/<staying>" -f "tag_list=<tag-a>,<tag-b>"

2. Move the declaration¶

In Nerthus.Infra/inventory/runners.yaml, delete the row under the old host and write it under the new one, keeping description equal to the new id and moving the slice numbers to what the new machine can carry (the two numbers). The vault file moves with it: vault/hosts/<new-host>/RUNNER_AUTH_TOKEN_<HOST>.sops.yaml.

nerthus-config plan --against live reads one delete and one create for a row whose host changed. That is the plan saying what it sees, and it is not what you want applied — the runner is not being replaced, it is being re-described. Apply the re-description yourself:

glab api --method PUT "runners/<id>" -f "description=<new-host>:<name>"

After that the live runner matches the new row by name, and the plan reads noop.

3. Down on the old machine, then converge the new one¶

systemctl stop gitlab-runner    # on the old host

One token, one queue, never two. Two supervisors holding the same token and answering the same queue is a state to pass through in zero seconds, not to schedule.

Then converge the new host (Apply from the laptop, or let its agent take the change):

nerthus-config cookbook apply ci-runner --host <new-host> --no-dry-run

Never re-provision from the token in config.toml

The supervisor rotates its own token and rewrites that file. Feeding the rotated value back is how both of this estate's runners once answered 403 to everything. If the token is lost, mint a new one with reset_authentication_token and put THAT in the vault.

4. Confirm the machine changed, not that the runner is up¶

glab api "runners/<id>" | jq '{status, architecture, platform, ip_address, tag_list}'

online is not the check. It was true of the machine you moved off, and if both supervisors were briefly up it was true of the wrong one. ip_address, architecture or platform is what proves the move.

Read what the new host is really running, which is the rendered file and never the declaration:

grep -E '^concurrent|memory =|cgroup_parent' /etc/nerthus/runner/config.toml

5. One real job, end to end, before you delete anything¶

Run a pipeline and read the job's runner, not the pipeline's colour:

glab ci run -R <project> -b main

For a move that also changes what the host can do, check the capability rather than the tag: an emulated build needs /proc/sys/fs/binfmt_misc/qemu-aarch64 with flags POF, and a host that lacks it is refused by the cookbook's own fact rather than failing inside a build.

Triage¶

Symptom	Cause
jobs sit `pending`, then `stuck_pending_no_matching_runners`	a tag names no online runner. Check `tag_list` on every runner against every tag the pipelines ask for
`plan --against live` shows a `create` and a `delete` for one runner	the description at GitLab is still the old id. Re-describe it; the plan is reading two resources because it can only see names
the runner is `online` from the new host but jobs land on the old one	the old supervisor is still up: `systemctl status gitlab-runner` there
the supervisor starts and immediately fails	`config.toml` still holds a placeholder — the token file was not on tmpfs when the cookbook ran. Converge again once the agent has decrypted it
jobs are killed for no reason a log explains	the slice is smaller than `concurrent × memory`, or the machine is smaller than the slice. `validate` refuses the first; the role's own fact refuses the second