Skip to content

The model artifact

Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: not yet written.

The language pipeline covers the two jobs in the annotation repository's pipeline — the scheduled one that writes nerthus.lang/, and the manual benchmark that measures what the pipeline costs on that runner. This page covers the two model jobs, frames-train and frames-yardstick, and where what they produce goes.

Both are Nerthus.Lang CI jobs, both are when: manual on the default branch, and both carry allow_failure. There is no schedule, no cron, and no retrain-when-enough-rows: a person presses them, and that is an operator ruling rather than a convenience. Adding a schedules rule reverses a decision rather than tuning one.

They live in the Nerthus.Lang project's own .gitlab-ci.yml, not in the template the annotation repository includes — so they are triggered from Nerthus.Lang's pipeline page, not from the annotation repository's.

The jobs, and which repository runs each

Seven jobs are defined across the two files, and the split by repository is the thing to read first — the lore side gets both jobs in the included template, whether or not a host asked for the second.

Job Defined in Where it runs Trigger Produces
nerthus-lang ci/lang.gitlab-ci.yml the annotation repository, via include: scheduled nerthus.lang/, as a merge request
nerthus-lang-benchmark ci/lang.gitlab-ci.yml the lore repository, via the same include: manual lang-benchmark.json, what each pipeline component costs on the runner
frames-train .gitlab-ci.yml Nerthus.Lang manual a trained artifact, as a job artifact
frames-yardstick .gitlab-ci.yml Nerthus.Lang manual the engage gate's two recalls and their controls

pipeline-smoke is a third manual job in Nerthus.Lang's own file and is a smoke test rather than an operator action; test and lint run on every pipeline there. The language pipeline owns both lore-side jobs, their variables and their runner.

Where a trained artifact goes

Nothing publishes it automatically. frames-train writes an artifacts/ directory, retained 90 days, and an operator downloads it into a host's .nerthus/models/artifacts/. The daemon's /model/* routes read it from there.

The layout the daemon expects under the repository's .nerthus/models/:

.nerthus/models/live.json                        current and previous, per surface
.nerthus/models/promotions.jsonl                 append-only, never rotated
.nerthus/models/runs.jsonl                       every training run a person asked for
.nerthus/models/artifacts/<id>/model/            the spaCy pipeline
.nerthus/models/artifacts/<id>/model.stamp.json  what the trainer stamped it with
.nerthus/models/artifacts/<id>/evaluation.json   what the training run measured

The artifact directory a job produces is complete on its own — the pipeline, the stamp and the evaluation report — so installing it is a copy into artifacts/<id>/. Publishing it to the generic package registry beside the Morfeusz wheel is the same gesture and the same precedent when a fleet needs it.

frames-config.json is a separate file with a separate job. It carries the engage gate and is not part of an artifact; the daemon looks for it at integrations.lang.frames_config or at <model root>/frames-config.json, and GET /model/evaluation refuses by name when neither resolves. See Configuration.

Promotion is not engagement. Pointing live.json at an artifact does not turn the model layer on for a consumer. The engage gate decides that separately and refuses on its own, so a wrong promotion cannot put a weak model in front of a reader.

Variables

Both jobs inherit the two the annotation and benchmark jobs also need, and both refuse by name rather than failing later — a refusal an operator can read is the whole point of the pattern.

Variable Needed by Refuses with
NERTHUS_LANG_MORFEUSZ_URL both, and every other job that loads the pipeline "refusing to run without a pinned dictionary — morfeusz2 is not on PyPI and the extra cannot resolve without it"
NERTHUS_LANG_SPACY_MODEL_URL both, and every other job that loads the pipeline "refusing to run without a pinned model"
NERTHUS_ANNOTATIONS_URL frames-train, frames-yardstick "refusing to guess where repozytorium-adnotacji lives — a script resolves repositories, never the operator's filesystem"
NERTHUS_GOVERNANCE_CORPUS_URL frames-yardstick "the gold cites posts in a thread capture that lives in no repository, and a comparison run against a corpus the key does not cite scores every row a miss"
NERTHUS_LORE_URL frames-yardstick "without a lore checkout every subject is unresolved… refusing rather than publishing a pair nobody can compare with the committed one"

Three more are job-level variables: rather than project settings, and are edited in .gitlab-ci.yml when a second surface is trained:

Variable Default What it selects
NERTHUS_MODEL_SURFACE governance-transitions the surface the artifact is for, and the key its engage-gate row sits under
NERTHUS_MODEL_CLASS governance-transition the sample-store class the rows are harvested from
NERTHUS_MODEL_CORPORA sprawozdania-capture the corpora a run accepts, declared before any row is read. Required rather than defaulted: a per-document split that mixes two corpora obeys the split rule and can still teach one corpus's voice to another's semantics

Two others are read by the module and belong to a local run rather than to CI: NERTHUS_LANG_MODEL_ROOT names a model store root when --store is absent, and NERTHUS_GOVERNANCE_EVAL_KEY names frames eval's answer key, which is not in the repository. NERTHUS_VALUE_EVAL_KEY is its counterpart for the value key.

frames-train, and what it will not do

It harvests only confirmed rows and stamps the artifact with the grade it trained on. Anything else is stamped as what it is, and the engage gate refuses it whatever the configuration says — so an artifact trained on measured rows cannot reach a consumer by misconfiguration.

The run is deterministic: --seed defaults to 40 and the job passes none.

Determinism is not comparability, and the seed picks the split

A different seed gives a different held-out set as well as a different model, so a held-out score is only comparable between two artifacts trained under the same seed. The daemon's ModelWouldRegress check refuses with RegressionIncomparable when the seeds differ rather than silently comparing two different questions. Because the shipped job passes no seed, every comparison it makes today is valid — and stops being the moment somebody passes one.

Measured 2026-08-12 on the nerthus-lang runner's environment (python:3.12-slim on aarch64): the pinned Morfeusz wheel installs in about 2 s, the spaCy model in about 69 s, and harvest plus train over the 85 confirmed rows then available takes under 2 s. The store now holds 312 confirmed rows (2026-08-12, repozytorium-adnotacji-dev fc46ed1), so a run today harvests more than that measurement covered; the install time is the part that dominates either way.

frames-yardstick, and why it is three runs

It scores frames-only against frames-plus-a-model at the event level, k-fold by document, so no model is ever asked about a document it trained on. The number it produces is what fills the engage gate.

It runs three times in one job, deliberately, and asserts the two controls rather than printing them:

  • --model none must reproduce frames-only exactly. If it drifts, the measurement between the arms means nothing.
  • --model every must move the number. If it does not, the seam carries no reach and a measured gain of zero cannot be told from a path that never ran.
  • --model trained is the measurement.

A control that lives in a separate manual job is a control nobody triggers, which is why all three are one press.

The gate that closes it

pyproject.toml carries addopts = "-q -m 'not models'", so tests/test_frames_eval.py is deselected by default, and selecting it with -m models skips it again unless NERTHUS_GOVERNANCE_EVAL_KEY, NERTHUS_GOVERNANCE_CORPUS and NERTHUS_LORE_ROOT all resolve. A skipped gate exits 0 and prints like a passing one, so frames-floors resolves all three from CI variables, runs with -rs, and then reads its own --junitxml: a run of zero gates, or of one skip, fails the job. It is neither manual nor allow_failure - a floor is a contract, and a contract nobody is obliged to satisfy is a comment.

The capture keeps its name

A document's key is the basename of the file it was read from, and every proposal id begins with that key. The job fetches the thread capture to SPRAWOZDANIA.md for that reason: under any other name all 121 ids move together, proposals_sha256 stops matching the fingerprint the floors recorded, and the gate reports a filename as though the extraction had regressed.

See also