Skip to content

Build the fleet image by hand

Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: none, retired with Core.

The package job did not produce an image, and a fleet still needs a digest to pin. This page builds one from a workstation and proposes the pin the job would have proposed. Set up pipelines owns the pipeline this stands in for; Fleet owns the upgrade wave a merged pin drives.

How you know you are in it

  • package is red on ci_quota_exceeded, or it never started because the branch is neither main nor multi-host-infra.
  • No deploy/<shortsha> merge request exists on the consumer fleet repo, and its versions.lock still pins the previous digest.

A hand build spends no CI minutes, which is usually the reason you are here. It also builds on whatever architecture the workstation is, which is what the two flags in step 1 are for.

Build and push

  1. Cross-build for amd64 from a module checkout at the commit you intend to ship, logged in to the registry:

    docker login registry.gitlab.com
docker buildx build --platform linux/amd64 --provenance=false \
    -f docker/Dockerfile -t registry.gitlab.com/margonem/nerthus/it/nerthus.core:main --push .

    --provenance=false is load-bearing

    Without it buildx publishes an OCI image index carrying the amd64 manifest plus an unknown/unknown attestation, and a digest that names an index makes docker select by host platform. Every arm64 host then answers no matching manifest for linux/arm64/v8 and stops converging. A single-platform manifest.v2+json digest is fetched as named whatever the host is, which is what lets one amd64 image run on an arm64 machine under emulation.

  2. Read the digest back and check what you published:

    docker buildx imagetools inspect registry.gitlab.com/margonem/nerthus/it/nerthus.core:main

    It must report linux/amd64 and one manifest, not an index. The digest it prints is the value the pin carries.

  3. Prove the image runs before you propose it. Pull it by digest on a device and let it boot against a lore clone: it should report the schema version versions.lock expects and answer /health. Nothing downstream re-checks it, and a merged pin reaches every replica before anyone looks.

Propose the pin

Run the same publisher the package job runs, from the module checkout, with the digest and the source commit:

./vm/Publish-NerthusImagePin.ps1 -SourceSha 61fa712 -WhatIf `
    -ImageRef registry.gitlab.com/margonem/nerthus/it/nerthus.core@sha256:85a125a52553d13586da5a20ef5e453863f49cd861684cbf2e005a7e8e71e4ed

It rewrites the consumer's image: line and its Built from … @ <sha> provenance comment and nothing else, opens deploy/<shortsha>, and stops. It reads the consumer path from NERTHUS_INFRA_PROJECT_PATH and pushes with NERTHUS_INFRA_PUSH_TOKEN, so pass -WhatIf first and confirm the rewrite it previews. Editing versions.lock in a merge request by hand is the same deploy; record in the comment above the line that this digest was cross-built, why, and what step 3 verified.

How you know you are out of it

The merge request is merged, versions.lock names your digest, and the hosts take it in wave order: replicas first, the primary primary_lag_minutes later. The sweep reports skew between the two while the wave runs, which is the wave working.

See also