Language pipeline

The language pipeline¶

Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: not yet written.

Nerthus.Lang annotates the corpus with Polish lemmas and named-entity offsets, storing the results under nerthus.lang/. Search uses these annotations but can still function on raw text alone. See the annotation sidecar reference for the file format.

The pipeline runs in CI.

Where the tree lives, and where the job runs¶

nerthus.lang/ is a repository of its own: margonem/nerthus/it/repozytorium-lingwistyczne. It is not in the lore repository any more.

The job runs in that repository's pipeline, and this is the shape to hold onto:

• its checkout is the output — the tree it writes;
• the two source corpora are cloned beside it, shallow and read-only: the lore repository and the transcript archive;
• the merge request opens against its own project, so the credential that opens it needs write on nothing else.

The alternative — keeping the job in the lore pipeline and pushing across repositories — was declined: it would leave the lore project scheduling writes it no longer contains, and holding a token that can write another project.

The keys did not change when the sources moved. A lore source keys as its path relative to the lore repository's root; an archived transcript keys as a literal nerthus.logs/ prefix plus its basename. Point each source-root flag at its corpus's own repository root and a split run emits byte-identical keys — which is why 2 207 committed sidecars did not have to be re-keyed.

Configuration¶

Set these variables on the annotation repository (repozytorium-lingwistyczne), not on the lore one:

Critical notes: - NERTHUS_LANG_REF is mandatory. The job refuses to clone an unpinned module. - NERTHUS_LANG_MORFEUSZ_URL is mandatory. The job refuses to run without a pinned dictionary — morfeusz2 is not on PyPI, so the module's [morfeusz] extra cannot resolve from an index. The job installs the wheel before the module, which is what lets the extra resolve. - The two source URLs are mandatory, and this is the refusal that matters most. Both corpus walks answer a missing directory by yielding nothing. An unset source URL would annotate zero files, report a clean run, and then hand prune a corpus with no sources — where every sidecar in the tree is an orphan and only the removal cap stands between the run and an empty repository. GitLab expands an undefined variable to the empty string, so empty is the state a missing setting produces, and that is the state the job tests for by name. - Read and write are two different credentials, and that is the split working. The job pushes to exactly one project and reads two others. One write-scoped credential in the clone URLs would let the annotation job push to the lore repository — the thing moving the tree out was meant to prevent. - The read side is a deploy token per source project (username nerthus-lang, scope read_repository, nothing else). GitLab Free has no project access tokens — which is why this estate reaches for personal ones elsewhere — but it does have deploy tokens, and a read-only clone is exactly the case that needs no person's identity behind it. Each is scoped to one project and revocable without touching anything else. CI_JOB_TOKEN is not used: its cross-project reach has to be granted on each source project's allowlist and re-granted at prod adoption. - The push side is an SSH deploy key, and the argument that said it could not be has been refuted. This page used to read: "the push side must be a personal token on the machine user, because a merge request has to be opened by an identity." The merge request is opened by git push options — ci/publish.sh:137-141 passes -o merge_request.create, .target, .title, .description and .remove_source_branch — not by an API call, so no personal identity is needed and the token's extra reach bought nothing. A deploy key is scoped to one repository by construction; a personal token on a machine account carries every repository that account can write. The narrower credential was available the whole time. - Do not leave NERTHUS_LANG_KIND empty. The job defaults to all, but an empty value will fail.

The annotation repository's .gitlab-ci.yml is one include:

include:
  - project: margonem/nerthus/it/nerthus.lang
    file: ci/lang.gitlab-ci.yml

The job uses resource_group: nerthus-lang to prevent concurrent runs on the same repository.

The second job in the same template: nerthus-lang-benchmark¶

ci/lang.gitlab-ci.yml defines two jobs, and the include: above brings both into the annotation repository. The second is nerthus-lang-benchmark: manual on the default branch, allow_failure: true, no schedule. It measures what each pipeline component costs on this runner, and it publishes lang-benchmark.json as an artifact retained 90 days.

It lives beside the annotation job rather than in Nerthus.Lang's own pipeline because it must run on the runner the annotation runs on — a cost measured anywhere else is a cost for somewhere else — and this is the pipeline that has that runner and the credentials to reach a corpus. It draws its slice from the cloned corpora, not from the checkout: the checkout holds sidecars and nothing else, and a slice drawn from it would be empty, which build_slice refuses by name rather than reporting as a very fast pipeline.

It shares resource_group: nerthus-lang with the annotation job, so the two never run at once — two model loads on a runner capped at 6144m is the measurement failing rather than the runner. It refuses by name on the same three variables the annotation job does (NERTHUS_LANG_REF, NERTHUS_LANG_MORFEUSZ_URL, NERTHUS_LANG_SPACY_MODEL_URL) and pins OMP_NUM_THREADS, MKL_NUM_THREADS and OPENBLAS_NUM_THREADS to 1, because a multi-threaded reduction is not order-deterministic and a benchmark whose thread count follows runner load is measuring the runner's other jobs.

Three job-level variables set the slice. They are edited in the template, not on the project, and they are the report's identity rather than tuning — a benchmark whose seed moves between runs measures a different corpus each time and its numbers cannot be compared:

Why it exists at all: a percentage is not a budget. The module's own figures are arm64 Darwin; this runner is amd64 Linux — runner 54792753, corrected from aarch64 on 2026-08-25 when the claim was read off the API rather than off the comment. Measurements on this page taken before that date were taken on aarch64 and say so where they appear. Only this job can say what the annotation costs where the annotation runs, and the peak RSS it reports is what the cap below is sized against. Work on Nerthus.Lang carries the figures it has produced so far and the local command that regenerates them.

The second runner¶

NERTHUS_LANG_RUNNER_TAG is deliberately a different variable from NERTHUS_RUNNER_TAG, so the two cannot be conflated: the per-job memory cap is runner-wide, and the shared runner's slots serve every project's pipelines. The annotation job needs a runner of its own — concurrency 1, answering the lang tag alone. Set up a self-hosted runner covers registering it as a second checkout.

How large, and on what evidence. The cap is NERTHUS_RUNNER_JOB_MEMORY and this runner sets it to 6144m. The binding workload is this job: the annotation's peak RSS was measured at 2,846 MiB once the dictionary is consulted for every content token. At 6144m that is 2.16× the measured peak.

3072m was the previous default and it is no longer enough. It left 7 % headroom over that 2,846 MiB across an architecture boundary, which is not headroom. A host still running 3072m will see the annotation job collect hard rather than fail, which reads as a slow job and not as an out-of-memory one.

Measured on the runner itself, 2026-08-21. This page used to say no figure described a full annotation run over today's corpus, and that until somebody ran one the cap was chosen on a peak measured elsewhere. Project 86 ran it, on amd64, over the whole corpus, and the figures are in The annotation on amd64 below.

The cap is a cap, not a reservation, so NERTHUS_RUNNER_CONCURRENT multiplies the worst case and not the resting one. On a laptop, check that product against Docker Desktop's VM; on a bare Linux host the machine's RAM is the ceiling and a slice is what bounds the sum (Set up a self-hosted runner). This runner runs concurrent = 1.

The annotation on amd64¶

Measured 2026-08-21 on the annotation runner itself, over the whole corpus, which is the run this page spent a release saying nobody had done.

annotated 2186   skipped 0   failed 0   chunked 12   seconds 1004.7
lemma_rows 105 974   name_rows 6 624   morph_rows 88 331   manifest_rows 2 218

peak RSS   1 790.0 MiB     sampled every 4 s with `docker stats`, against a 6144m cap

The 2 846 MiB this page has been quoting was taken on arm64 Darwin with 14 cores. On the runner, on amd64, the same job peaks at 1 790 MiB — so the margin at 6144m is 3.4×, not the 2.16× computed from the old figure, and 4096m would still be 2.3×. That is not a recommendation yet: one run is one sample. It is written down against the cap that was in force, which is what makes it quotable at all.

It is slower than the same job on the laptop — 1 004.7 s of annotate time against 424.7 s. That job was never emulated (python:3.12-slim is a multi-arch index), so the comparison is eight cores the box owns against fourteen the laptop shared, and this workload prefers the fourteen. Recorded as measured rather than explained away.

Swapping the wheel changed build and nothing else¶

The move replaced a hand-built manylinux_2_28_aarch64 wheel with SGJP's official manylinux_2_28_x86_64 one. Same library 1.99.15, same dictionary 20260601, different compiled binary. Comparing the two runs' trees sidecar by sidecar, key by key:

The prediction was earned rather than hoped for: build is a digest of the compiled libmorfeusz2, this wheel and SGJP's are two of the three builds already compared, and build is deliberately outside the skip check. It held exactly.

The 19 are all Nerthus_* sidecars whose source documents left the lore repository between the two runs, with the public-website split. The pruner removed the orphans, which is what it is for — checked against the lore repo rather than assumed.

Model pinning¶

The pipeline installs the model from NERTHUS_LANG_SPACY_MODEL_URL. A model bump rewrites the entire corpus, so it must be a deliberate, reviewed change. Treat it like any other major update.

Locally, use python -m spacy download pl_core_news_lg for development. In CI, always pin the wheel to ensure reproducibility.

The cache key (lang-$NERTHUS_LANG_REF) ensures a version bump invalidates the cache, forcing a fresh model download.

The Morfeusz wheel is pinned the same way, and since 2026-08-21 it is SGJP's own:

https://download.sgjp.pl/morfeusz/20260601/Linux/manylinux_2_28/64/
  morfeusz2-1.99.15-20260601-cp310-abi3-manylinux_2_28_x86_64.whl

The sentence this paragraph used to carry reverses on amd64. It said: SGJP publishes amd64 Linux wheels only and the annotation runner is aarch64, so ci/build-morfeusz-wheel.sh builds the wheel from source. The annotation runner is amd64 now, the upstream wheel fits, and the estate stops maintaining a from-source build of somebody else's library. This is the one part of that move that makes the system simpler.

ci/build-morfeusz-wheel.sh stays, and so does its history. The operator's laptop is still aarch64 and still carries a runner (the arm64 half); the script is the record of how that wheel was made and of what its platform tag was retagged away from. If an aarch64 annotation runner is ever wanted again, it is what makes one.

The dictionary date in the wheel becomes morfeusz_dict in every sidecar's pipeline stamp, so changing it re-annotates the whole corpus. It did not change here — 20260601 either way. What changed is the compiled binary.

The wheel's identity and the dictionary's are two different things, and only one of them re-annotates. From format 7 every sidecar also carries a build field — a digest of the compiled libmorfeusz2 and its extension module — because the dictionary date does not distinguish two builds of it: this project's wheel, the SGJP amd64 wheel and a Darwin install all report 1.99.15 and pl.sgjp.sgjp-2026.06.01, and they do not all enumerate a word's readings in the same order. build is recorded, published as the fifth column of index.tsv, and deliberately not part of the skip check, so rebuilding the wheel does not rewrite the corpus. Changing the dictionary date still does.

Run output¶

Each run produces a lang-report.json artifact (retained for 30 days) and posts the run report as the merge request's description, compacted to a single line — git refuses a push option containing a newline, so a pretty-printed report would kill the publish of a complete, correct run. The pretty copy is the job artifact. The report's fields:

The last three are the row counts of the corpus TSVs, which every run rebuilds from the tree — including a run that annotated nothing — so the two dictionary files and the manifest always describe the sidecars on disk.

The gazetteer, and the run that has to rewrite everything¶

Before annotating, the job reads the lore checkout's entity store and turns it into matcher patterns that run ahead of the statistical component. It prints a one-line summary to the job log — entry counts by origin, pattern count by label, and the digest — and refuses the run with exit 2 when it builds no patterns at all, because an empty pattern set annotates exactly as the layer did before it existed and would otherwise report a clean, silently worse corpus-wide run.

Two flags govern it. --entity-index names the store, defaulting to nerthus.entities.md under the lore root. --no-gazetteer means the degraded run on purpose; the stamp then carries no gazetteer field, which is itself a corpus-wide invalidation in the other direction.

An edit to the entity store rewrites the corpus, and that is deliberate. The pattern digest is inside the pipeline stamp, so adding one @alias row invalidates every sidecar and the next run re-annotates all of them. A full rewrite costs about 10 MB of permanent pack — the figure is in the repository README, measured — so an operator adding a batch of entity blocks should expect the run after it to be a full one, and should batch such edits rather than trickling them.

The bridge rides with the patterns (format 9, 2026-09-03). The same entity store is also cut into a stem table, applied after the ruler and the tagger: a written form is accepted only when it is one closed Polish ending away from a declared name — klasztoru Różanitów, Elancji — and spanned with the index's label and the heading as lemma. Nothing is generated; the corpus wrote the form and the index anchors it. A name-derived adjective or demonym (Ederskich, thuzalskim) is labelled deriv, NKJP's class for a derivative, which the sidecar carries and names.tsv does not. The table is part of the gazetteer digest, so it moves on an index edit and on nothing else. The same run refuses link plumbing as a span (Werbin.md, a URL, ](, a field label) and reads a session heading by its grammar: one date span, the narrator field a person, Rada there nothing. --no-gazetteer switches the bridge off with the patterns.

The parser is on since format 10, for one consumer. mentions.py bounds a nominal group by the parse and publishes it as a candidate when its head is a role, creature or group lemma from the pipeline's lexicon — the shape the ruled frame found the tagger silent on. It costs +13 % of the run (measured on 33 texts); --no-mentions loads no parser, writes an empty mentions array and drops the mentions digest from the stamp, which rewrites the tree the next time the lexicon is back.

A source longer than 250,000 characters is chunked, not refused: it is cut at boundaries the document declares — a Markdown heading, then a blank line, then a line break, then sentence-final punctuation — and each piece is annotated separately with its offsets shifted back into the source's own coordinates, so no sentence is split and every span still indexes the source. The chunked field counts them. The twelve largest sources in the development corpus — player character sheets and per-town session logs — are annotated this way.

Exit codes: 0 (success), 1 (partial failures), 2 (failed to start). A single failed file won’t stop the run.

After annotation, the job runs prune to delete sidecars for deleted sources, with the same source roots the annotation used. Passing them is not optional: prune decides what to delete by walking for sources, so a prune pointed at the output checkout alone finds none and condemns the whole tree.

Prune refuses a removal larger than 100 sidecars and exits 2, failing the pipeline, because a lore directory rename looks identical to mass deletion from the prune walk. The triage is to read the would-be-deleted list in the job output — and if it names most of the corpus, the fault is a source root, not a rename. Re-run with --max-removed 0 only once the diff has been read and the roots have been confirmed.

Publishing¶

The pipeline creates a merge request. It only modifies the nerthus.lang/ directory, so even a misconfigured run can’t corrupt other files.

The annotation repository must carry these .gitattributes rules, and they had to be there before the first sidecar was committed — attributes decide how blobs already in history are compared, so a tree committed without them is not repaired by adding them later. They travelled with the tree out of the lore repository:

nerthus.lang/** -text linguist-generated
nerthus.lang/** -diff
nerthus.lang/*.tsv -text linguist-generated diff

These rules: - Prevent false "modified" status for JSON files (-text) - Exclude sidecars from diffs (-diff) - Keep TSV files diffable (last rule overrides the first two)

Licenses¶

The model IS redistributed, and this page used to say it was not. It read "only used in CI to generate data—not redistributed". Measured 2026-08-25: nerthus.lang (project 84956243) is visibility: public, its generic package registry serves pl_core_news_lg-3.8.0-py3-none-any.whl — 573 718 074 bytes — anonymously, and NERTHUS_LANG_SPACY_MODEL_URL points CI at that mirror rather than at upstream. Operator gate H94.4 decided the document moves rather than the practice, because the practice is defensible: the mirror is a verbatim, unmodified copy of a wheel explosion/spacy-models already publishes publicly, with the GPL-3.0 text and upstream attributions intact inside it.

The claim carries its own probe rather than asserting a state that ages:

curl -sSI https://gitlab.com/api/v4/projects/84956243/packages/generic/pl-core-news-lg/3.8.0/pl_core_news_lg-3.8.0-py3-none-any.whl

Expect 200 with no credential. A 404 or 401 means this row is out of date and the CI fetch is broken in the same moment. Nerthus.Lang's own README carries the same note beside the mirror's sha256.

The lemma authority is Morfeusz, not the model’s lemmatizer.

Limitations¶

• Proper nouns are never handed to the dictionary. Morfeusz declines a name like Martel or Erastera wrongly, so the tagger’s own lemma stands instead and a sidecar may carry a guess at a citation form. Annotation sidecars owns that field contract; declining names is name resolution’s job.
• Prose only. Indexes, contributor stores, and ledgers are left untouched.

See also¶

• Annotation sidecars
• Search
• Work on Nerthus.Lang
• Set up a self-hosted runner
• Set up GitLab pipelines