Skip to content

Recover a failed commit-back

Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: none, retired with Core.

Before you start: where the evidence is

The push this page is about is recorded in the daemon's operational.jsonl, inside the lore volume — not in docker logs, which carries one boot line and nothing else. Read the logs is the map.

fleet:provision or fleet:rotate changed Cloudflare and then failed to publish what it did. The Cloudflare side stands; secrets/, rendered/ and state/ in the fleet repository do not. This page is that repair. Fleet owns the converge loop the unpublished state feeds.

How you know you are in it

The job is red on its last step, and the message names the commit-back. There are three of them:

The job says What happened
commit-back could not rebase onto main — conflicts in: … Something else pushed while the job ran, and the rebase collided
commit-back pushed <sha>, not the commit this run made (<sha>) The push reported success on a commit that was not this run's, so the render it carried is gone
commit-back failed after 3 attempts Three pushes were rejected in a row

An expired NERTHUS_INFRA_PUSH_TOKEN reaches the same place from the other direction — see Rotate secrets and re-mint it before re-running anything.

Which job failed decides how urgent this is:

  • After fleet:provision, hosts converge onto a descriptor and a ciphertext that never arrived, so they keep running what they already run. Nothing gets worse while you work.
  • After fleet:rotate, the old tunnel token was invalidated at Cloudflare the moment the secret was re-minted, and the new ciphertext never reached git. That host's connector is down and stays down until this repair lands.

The repair

  1. Read what actually landed. On the default branch, git log --oneline -3 -- secrets rendered state. A chore: fleet provision state [skip ci] commit newer than the failed run means the push did land and the failure is somewhere else in the job.

  2. Settle the branch. Merge or land whatever arrived while the job was running. The conflict is always in one of the three paths the run rewrites, and the next run regenerates all three, so nothing in them is worth merging by hand.

  3. Re-run the job on the default branch. Provision is idempotent, and a second rotation costs no more than the first: the token it replaces was already dead.

  4. Confirm. secrets/<town>.tunnel.age carries a commit newer than the failed run, state/fleet.lock.json moved with it, and Get-NerthusFleetState.ps1 -Repo . -GitLab no longer reports provision-failed. On a rotation, the host's hostname answers again within one agent tick — five minutes.

Do not

Do not hand-write ciphertext or hand-edit state/fleet.lock.json to close the gap. The ledger is what the next provision reads back against Cloudflare, and a record nothing at Cloudflare matches fails the ownership read-back on the run after.

See also

  • Rotate secrets — the push token this job spends, and the rotation the fix re-runs
  • Recover a diverged host — the reader that reports provision-failed and ciphertext-suspect
  • Fleet — what the unpublished descriptor would have told the hosts