Work on Nerthus.Lang¶

Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: not yet written.

Nerthus.Lang is the one Python project in the ecosystem: it annotates the lore corpus with Polish lemmas and entity spans and writes annotation sidecars, and it runs the frame grammar that proposes governance transitions. This guide is how to run it and its gates on a laptop. The CI job that runs it for real is the language pipeline.

Set up¶

python -m venv .venv
.venv/bin/pip install -e ".[dev]"
.venv/bin/python -m pytest

The suite needs neither model, and that is the design rather than a shortcut. The disambiguation rule is a pure function, the corpus walk reads Markdown and JSON, and run() takes its annotator as an argument, so a test substitutes a trivial one. Requires Python 3.11 or newer; CI runs python:3.12-slim.

Survey a checkout before annotating it¶

.venv/bin/python -m nerthus_lang survey /path/to/lore-checkout

survey loads no model. It reports file and character counts per half of the corpus, the output directory, and the unusable transcripts by name.

Measure what each component costs¶

.venv/bin/python -m nerthus_lang benchmark /path/to/lore-checkout --seed 40

benchmark draws a seeded slice of the corpus and times the pipeline per component, one process per variant, so "is the parser expensive" is a question you answer rather than quote. --kind narrows to lore or transcript, --json and --out emit the report as JSON.

Run it before repeating any runtime figure, including the ones below, because the answer is platform-dependent and this is the whole reason the command exists.

The report carries two instruments and they are not the same number. Ablation re-runs the whole pipeline with one component removed and compares wall clock; share divides one component's own timer by the accounted total. A component can be 15.8 % of the accounted time and cost 21.9 % to enable, because removing it also removes work downstream of it. The tables below are split on that line, because a two-row table invites reading down a column and the source distinguishes the two.

Both readings come from the benchmark command's own JSON report, over a slice the report names: seed 55, 60 documents, 980 944 characters, 195 642 tokens, three repeats, one process per variant, drawn from repozytorium-fabularne-dev at 49fc214.

Ablation — the pipeline re-run without the parser:

Share of accounted time, over the same 195 642 tokens:

Both platforms are published because either one alone would be quoted as if it were the number. The parser costs about 1.6× more where the job runs than where it was first measured, on both instruments, and every reading refutes the claim they replaced — that the parser was "roughly half the runtime", a figure that entered at the initial import with no benchmark behind it. The two platforms differ in operating system, architecture and Python minor version at once, so neither is a controlled measurement of architecture alone; what they establish is that the number moves, not by how much any one difference moves it.

Runtime is not why the parser is excluded, which the module's own docstring says before it gives these figures: the exclusion rests on the parser not being output-neutral, and pipeline.py carries that measurement.

Annotate for real¶

Both models, roughly 500 MB, and a GPL-3.0 licence on the spaCy one — see the licence table before you redistribute anything.

morfeusz2 is not on PyPI, so requesting the extra on its own ends in No matching distribution found. Install the SGJP wheel by URL first and the extra then resolves against the installed wheel. The date in the wheel URL is the dictionary release and lands in every sidecar's stamp as morfeusz_dict.

ci/build-morfeusz-wheel.sh is no longer part of the pipeline and is not what you want here. It built an aarch64 wheel from source because SGJP publishes none and the annotation runner used to be aarch64. Runner 54792753 is amd64, so the upstream wheel fits and CI installs SGJP's own x86_64 build — mirrored into nerthus.lang's package registry and hash-verified before install, file_sha256 089a83ab03a137a57e23d9d42028d80b8858d1a4de78f1db86a18ba504400a98. The script is kept only because the aarch64 wheel is still in that registry, nothing else records how it was made, and an aarch64 annotation runner may be wanted again — see the language pipeline.

Your wheel is almost certainly not CI's, and the sidecar records which build wrote a document — in index.tsv column 5, not in the sidecar body, since format 8. build is a digest of the compiled library and differs between this wheel, a Darwin install and the x86_64 wheel CI uses even though all three report the same version and the same dictionary. It is recorded and not compared, so annotating locally and pushing does not make CI rewrite the tree — and the two trees agree on their lemmas anyway, which they did not before 7. It moved out of the body in format 8 because a host change was rewriting 1 633 sidecars to change nothing but that field, at ~9.9 MB of permanent pack per move.

.venv/bin/pip install http://download.sgjp.pl/morfeusz/20260601/Linux/manylinux_2_28/64/morfeusz2-1.99.15-20260601-cp310-abi3-manylinux_2_28_x86_64.whl
.venv/bin/pip install -e ".[morfeusz]"
.venv/bin/python -m spacy download pl_core_news_lg
.venv/bin/python -m nerthus_lang annotate /path/to/lore-checkout --limit 20

--max-chars is the largest piece the annotator is handed at once. A longer source is cut at a boundary the document itself declares, each piece annotated separately, and every span shifted back into the source's own offsets; the report counts such a file under chunked. The language pipeline names the boundaries the cut tries, in order.

prune deletes sidecars whose source is gone. Run it after deleting lore locally, and know that CI runs it on every annotation job — a deleted character file otherwise leaves an annotation no source hash will ever invalidate. It counts before it deletes: a removal above --max-removed (the default is 100) is refused with exit 2, printing would_prune, the doomed file list, and the cap. To the walk a directory rename is indistinguishable from deletion, so read the diff before re-running with --max-removed 0.

The run writes into nerthus.lang/ under the positional root and nowhere else. It never touches the lore, the archive, or .nerthus/. Against a single checkout holding everything, that root is also where the sources are read from; --lore-root and --transcript-root name them separately, which is how CI runs it — output in the annotation checkout, sources in two read-only clones. Pass the same roots to prune: it decides what to delete by walking for sources, so a prune without them condemns the whole tree.

Draw the evening's questions¶

.venv/bin/python -m nerthus_lang propose /path/to/transcripts-checkout \
  --lore /path/to/lore-checkout --out proposed.jsonl --per-source 3

For every archived transcript that exactly one lore session heading links, propose turns the format-10 sidecar's mentions into candidate rows: the capitalised nominal groups outside any span, deduplicated by lemma and capped per transcript, scoped to that heading verbatim, with the transcript's resolved speakers as candidates and a referent only when the cast is one person. A transcript two headings link, a sidecar older than format 10 and a sidecar whose hash no longer matches the archive are skipped and counted in the summary line. The JSON Lines go to the annotation store's tools/backfill_123_proposed_questions.py, which deposits them at grade proposed; the dashboard's closing drawer reads them by scope. Nothing here writes a store, a sidecar or the lore.

Run the frame grammar¶

frames is a subcommand tree of its own, and it needs the parser the annotation pipeline excludes. It loads it itself; annotate is untouched.

.venv/bin/python -m nerthus_lang frames propose SPRAWOZDANIA.md \
    --out proposals.json --lore ../repozytorium-fabularne

--lore is optional and costs something when it is absent: without a checkout no subject resolves, and every confidence sits one rung lower. The command says so on stderr rather than in the file.

eval needs the answer key, which is not in this repository — set NERTHUS_GOVERNANCE_EVAL_KEY to its path. It exits 1 when a floor goes red, which is a result rather than a crash.

coverage takes the same output file and a ledger directory, and answers the question the consumer cannot ask about itself — what did the machine report that I never used? It exits 1 while anything is unread, for the same reason eval does.

.venv/bin/python -m nerthus_lang frames coverage proposals.json \
    --ledger ../repozytorium-dziel/governance/ledger

Run it after any pass that builds something out of transitions. A consumer measuring only how much of its own work the proposer agrees with has measured one direction of a two-directional comparison, and it is the direction that cannot find a miss.

train writes an artifact stamped with the grade it trained on. Only confirmed rows produce a model any consumer may see; anything else is stamped as what it is and the engage gate refuses it whatever the configuration says.

yardstick is what fills the engage gate's two recalls, and it is the only comparison that may: it scores both configurations at the EVENT level, k-fold by document, so no model is ever asked about a document it trained on. It takes the corpora it accepts the way train does, and it carries its own two controls — --model none must reproduce frames-only exactly and --model every must move the number. Run both beside any measurement; a gain nobody bracketed that way cannot be told from a seam that never ran. Governance proposals explains what it reports and why the parts are never folded into one figure.

.venv/bin/python -m nerthus_lang frames yardstick ../repozytorium-adnotacji SPRAWOZDANIA.md \
    --corpus sprawozdania-capture --lore ../repozytorium-fabularne --out comparison.json

Read the exit code¶

One unparseable file never ends a run. The report on stdout carries annotated, skipped, failed, junk_sidecars, failures and seconds; the pipeline keeps it as lang-report.json and pastes it into the merge request.

Know the gates¶

Four gates fetch: frames-floors, pipeline-smoke, frames-train and frames-yardstick clone the annotation store and the lore repository, and two of them also curl the thread capture. python:3.12-slim carries neither git nor curl, so each of the four opens with !reference [.tools-clone, script] and installs both. Leaving that line out fails late and expensively: the job installs Morfeusz and the spaCy model for six minutes, then dies on git: command not found at its first clone. The install stays in the jobs that fetch rather than in the base image, for the reason the test job gives - python:3.12 carries both and costs a gigabyte, and a line in a job says which gate depends on fetching.

pipeline-smoke selects the tests marked models, which pyproject.toml excludes by default through addopts. It downloads hundreds of megabytes, so it stays off the fast path and needs NERTHUS_LANG_MORFEUSZ_URL and NERTHUS_LANG_SPACY_MODEL_URL set on this project as well as on the annotation repository (repozytorium-lingwistyczne, whose CI runs the scheduled annotate job) — the job installs the Morfeusz wheel before the extras and refuses to run when either variable is unset. Run the same selection locally with -m models once both models are installed.

The two manual jobs are operator actions and are allow_failure, because what they produce is a number somebody reads before deciding — a red pipeline is not how that decision should arrive. Both refuse by name on every input they cannot resolve rather than substituting one: frames-yardstick additionally needs the thread capture, which lives in no repository, and it refuses instead of scoring against a corpus the gold does not cite. Its control step asserts that --model none reproduced frames-only and that --model every moved the number, and fails naming which one broke.

Ruff waives N802, N803 and N806. PascalCase locals are the house convention across every Nerthus tree, and Code style beats PEP 8 inside one repository.

Do not lemmatize a proper noun¶

CONTENT_POS includes PROPN, and the dictionary is deliberately kept away from it. On a sample of the entity index Morfeusz turned Martel into Martela, Drzewiec into Drzewce, and Karka into karko — a surname inflected wrongly, a village renamed, and a nonsense common noun. Declining invented names belongs to name resolution, which is built for exactly the words a dictionary lacks. mieście reaches miasto; Solmyrze never reaches Solmyra here.

See also¶

• What the language layer is for — the subject, and what it deliberately cannot answer
• Annotation sidecars — the output contract this code satisfies, and the seven readers of it in Nerthus.Core
• Governance proposals — the frame grammar's output contract and its engage gate
• The language pipeline — the scheduled annotation job, its variables, and its runner
• The model artifact — the other two CI jobs, their variables, and where a trained model goes
• Search — the largest reader, and what a lemma buys a query