Roll the fleet back¶
Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: not yet written.
When a pinned image turned out to be bad and nothing is lost.
This is not Restore the estate. Nothing is missing; the fleet is running the wrong thing. Sending both incidents to one page serves neither.
The clock¶
Nerthus.Core.Infra.Dev/fleet.settings.yml sets primary_lag_minutes: 15. A bad pin reaches both
replicas at once and the primary a quarter of an hour later.
The estate can break itself in 15 minutes and can un-break itself for about 28 hours. Measured 2026-08-26: of the 86 digests ever pinned, 7 resolve. A bad pin found on Monday cannot be reverted on Wednesday — only rebuilt, from a source tree that has moved on.
So: check whether the target still exists before you plan around it.
TOKEN=$(curl -s "https://gitlab.com/jwt/auth?service=container_registry&scope=repository:margonem/nerthus/it/nerthus.core:pull" | jq -r .token)
curl -sI -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $TOKEN" \
-H 'Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json' \
"https://registry.gitlab.com/v2/margonem/nerthus/it/nerthus.core/manifests/<digest>"
Probe the digest the fleet is running now in the same loop.
Without the manifest-list Accept headers a perfectly live OCI index answers 404. If your
current pin also reads 404, the probe is broken — wrong token, wrong headers, no egress — and
it is telling you nothing about the target. This is the single most likely way to conclude that
a good rollback is impossible.
The operation¶
fleet:rollback is a manual job on the consumer repo's default branch. It reads image_previous:
out of versions.lock, probes the target before it writes a byte, and commits the swap.
It is a swap, not a one-way move: after rolling back, image_previous: names the digest you
rolled away from, so rolling forward again is the same operation.
If versions.lock carries no image_previous:, the tree predates that field. The digests are in
git log -p -- versions.lock; whether their bytes survive is the probe above.
The lag works against you here, on purpose¶
The soak lag exists so a bad image reaches a replica — which cannot publish — before it reaches the primary, which can. During a rollback the same lag delays the repair of the host that is actually serving.
This was decided rather than inherited: the lag stays, because inverting the wave means the converge agent taking a code path that has never run, at the worst moment to find a bug in one. The job prints the wall-clock time the primary will converge.
To repair the primary now, on that host:
Why a superseded image usually still exists now¶
CI pushes the merged manifest to $CI_COMMIT_REF_SLUG, which is main — a mutable tag, so every
superseded manifest lost its only tag on the next build and was collected. Since 97/02,
Publish-NerthusImagePin.ps1 also applies an immutable pin-<12 hex> tag at pin time, so a pin
whose image has no tag is impossible by construction rather than by policy.
Pins written before that change have no such tag. Check the probe, not the calendar.