Skip to content

Test a cookbook

A cookbook is tested three ways, cheapest first. Each arm proves something the one before it cannot, and none of them is a real host: a real host is converged by its own agent, after the arms below have passed.

Arm Where Proves Cannot prove
Unit cargo test, anywhere every step type's check and apply, for each of the four families, against an in-memory host; idempotence that apt, apk, dnf, systemd and OpenRC behave as the step assumes
Machine container an Incus container on the lab host the role converges on the family's real image, and a second run changes nothing anything the container shares with the lab's kernel (below)
Virtual machine an Incus VM on the lab host the same, on the family's own kernel what differs between a VM and the rented machine (disk, provider network)

The unit arm

crates/nerthus-config-core/tests/cookbook_steps.rs runs every step type against a fake host of each family: plan, apply, plan again, apply again. The first plan must hold a change, the second must be all ok, and the second apply must carry out nothing. The fixtures differ where the families do (each pins another version of the same package, the Ubuntu host carries the provider's 50-cloud-init.conf sshd drop-in), so a step that is right for one family and wrong for another fails here rather than on a host.

The lab arms

uv run nerthus-config test cookbook trivial --family alpine \
    --infra ../nerthus.infra --cookbooks tests/fixtures/cookbooks \
    --values tests/fixtures/lab/values.yaml

The harness runs from a laptop or from the site worker, never from CI, which holds no host key. It uses only the host whose inventory row carries the lab role, and refuses any other by name. It:

  1. launches nc-test-<role>-<family> (-vm with --vm) from images:<family>/<release>, the release the family's pins name, deleting any instance of that name first;
  2. waits until the instance answers incus exec and resolves the archive's host name;
  3. runs the cookbook twice over one shared SSH connection through incus exec;
  4. writes first.json, second.json, facts.json and result.json under --out, and deletes the instance unless --keep is given.

It exits 0 when the second run carried out nothing and neither run failed. result.json carries the facts hash: the SHA-256 of everything the second run read, with each file as its digest, never its bytes. The same role on a fresh instance of the same image gives the same hash, which is what a sprint's drill compares.

Two families need an option to boot as a VM, and the harness passes it: AlmaLinux's image asks for an agent:config disk, Alpine's refuses secure boot. The lab host needs qemu-system-modules-spice for any VM to start.

What a container cannot prove

A container shares the lab's kernel. So every check whose subject belongs to the kernel rather than to the filesystem needs the VM arm. Measured on the lab:

  • Time. chronyd does not start in a Debian, AlmaLinux or Alpine container (adjtimex … Operation not permitted): a container may not set the host's clock. Ubuntu's package started it anyway. A step asserting that chrony runs cannot converge where it is refused, so a role enables it in the container arm and asserts running in the VM arm, where the Debian and Ubuntu VMs ran it.
  • The kernel itself. A container reports the lab's kernel; a VM reports the image's own.

By the kernel's design, and so in a container never tested at all:

  • sysctls the kernel does not namespace (kernel.*, vm.*, most of fs.*); a container can set only its own network namespace's net.*;
  • kernel modules, binfmt_misc, swap and the boot command line;
  • the mandatory access control policy: an AlmaLinux container on the Debian lab has no SELinux of its own, so an enforcing check is a VM check.

Every check in the CIS catalogue therefore names the arm that proves it, container or vm; a check whose subject is in the lists above is vm.