Annotation sidecars¶

Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: not yet written.

The on-disk contract between Nerthus.Lang and everything that reads it. One JSON file per annotated source, committed under nerthus.lang/ in repozytorium-lingwistyczne — its own repository since August 2026; a host finds the tree through corpora.lang, falling back to the in-lore copy of an un-migrated checkout (Configuration) — produced by the language pipeline in CI. Nerthus.Core reads these files and never runs the pipeline.

Where a sidecar lives¶

The path is derivable from the source key by string manipulation alone: fold / to _ and append .json. The directory is flat, one level deep, the same layout nerthus.logs/ uses.

A transcript's sidecar name carries .json.json because the source is itself a JSON file. Nothing strips the source extension — a reader holding the source key needs no table to find the annotation.

Fields¶

source_hash is taken over the decoded text so a line-ending normalization on checkout does not read as a content change on one contributor's machine and not another's.

pipeline¶

build LEFT THE SIDECAR BODY AT FORMAT 8 and is now written to index.tsv column 5 alone. It used to sit here as a string. It was rewritten into every sidecar on every host change — the arm64 to amd64 move rewrote 1 646 of them, of which 1 633 differed in this field and nothing else, at 9 935 936 bytes of permanent pack in a repository every daemon container clones, three times in a fortnight. A reader older than 8 asks the body for build, finds nothing, and reports what the file says rather than a claim about some build — the same answer read_manifest already gives for a four-column row written before 7. Everything below about what the field MEANS is unchanged; only where it is stored moved.

morfeusz_dict does not identify a Morfeusz install, which is what build is for. Three builds were measured against one another — a Darwin arm64 install, the PyPI morfeusz2 wheel on amd64, and the aarch64 wheel NERTHUS_LANG_MORFEUSZ_URL names, which CI uses. All three report library version 1.99.15 and dictionary pl.sgjp.sgjp-2026.06.01, and the third returns its analyses in a different order from the other two. So neither the version nor the dictionary id distinguishes a build, and before format 7 a tree could not say which one wrote it. build is a digest of the compiled binaries — libmorfeusz2.* and _morfeusz2.* — because they are the only thing available at runtime that differs.

Versions, not names. A name resolves to whatever is installed, and a run that cannot say which revision it used cannot tell a lore edit from a model bump later. "format": "tagger" is how a reader detects a degraded artifact: Morfeusz is the lemma authority, and without it the tagger's own lemmas stand — pl_core_news_lg ships trainable_lemmatizer, an edit-tree model with an orth backoff, and it is the weakest component in the pipeline at 94.19 % against 97.83 % for coarse part of speech. Both figures are the model package's own meta['performance'], on the model's own evaluation set and not on this corpus — no gold labels of any kind exist here, which is why they are the model's claim about itself rather than a measurement over the lore.

The library and the model data get a field each because they move independently: data can be retrained under a fixed library, and the library can move under fixed data. Tokenization belongs to the library, so spacy is the field every offset in this file follows. A run compares the whole pipeline object against the recorded one, so a change to either version invalidates every sidecar at once and the next run rewrites the tree.

lemmas[]¶

Three-element arrays, [folded surface, lemma, count], sorted.

A projection, not a token stream. Format 3 wrote one record per content word with its offsets and part of speech, and nothing in Nerthus.Core ever read start, end or pos — every reader of this array folds it into one corpus-wide dictionary, so the offsets and parts of speech had no consumer anywhere.

The projection today, measured 2026-08-12 over nerthus.lang/ in repozytorium-fabularne-dev at 49fc214: 2 207 sidecars carry 1 096 386 triples, standing for 1 654 620 token attestations, and the lemmas array serializes to 27.68 MB of a 51.73 MB sidecar tree — 53.6 % of it. The format-3 tree it replaced was measured at 1 826 834 token records at 139.78 MB, about 84 % of that tree, during the format-3-to-4 migration. That tree no longer exists and the figure cannot be re-derived; it is recorded here as what the projection was decided on, not as a fact about the corpus today. The share fell because spans grew, not because the projection shrank.

Kept per document rather than only in the dictionary because a run is incremental. A run skips a source whose hash and stamp still agree, so it never sees that source's tokens again, and a dictionary rebuilt from only what the models ran over would lose every skipped document. Rebuilt from the sidecars, it is exact after every run.

Only content words are recorded: NOUN, PROPN, VERB, ADJ, ADV, NUM. The full token stream is roughly 30 % punctuation and markup, and no index queries w or się.

A pair whose surface folds onto its own lemma is still written, with the folded surface in position 1. It cannot rewrite a query — the reader would look the surface up and find the key it already had — but it is an attestation, and attestations decide contested surfaces. karka is attested 2 608 times as its own base form and 611 times as karko; dropping the first group made karko the winner and had the index rewrite a base form into an inflection.

URLs are not tokens. The tagger calls them adjectives and lemmatizes them to themselves, and the session registries are mostly links. Nothing queries a URL as prose.

Which lemma one token gets, and when it gets none. Morfeusz returns every reading a surface admits and the tagger's part of speech picks among them. Where exactly one distinct lemma carries that part of speech, it is published. Where several do — and that is the ordinary case, not the exception; Bramy under NOUN has eight readings — the tag has decided nothing, and spaCy's own lemma stands instead. The dictionary is not asked to break a tie it has no way to break.

Before format 7 the pipeline took whichever reading Morfeusz returned first, and that order is a property of the compiled library rather than of the dictionary the stamp records: three builds were measured, all reporting 1.99.15 and pl.sgjp.sgjp-2026.06.01, and one of them enumerates differently. Two hosts therefore published different rewrites for the same corpus — 132 rows of lemmas.tsv and 59 of names.tsv, with bramy polnocy reading Bram północ on one and bram północ on the other over 88 attestations. Ordering the candidates instead would have kept every row and decided Bram against bram by codepoint, which is an alphabetical verdict on a name.

The population, measured over the corpus at 59e75dc9: of 2 311 978 lemma decisions, 80.9 % are settled by the tag, 11.1 % match no reading at all, and 8.0 % — 185 550 — are withheld because several distinct lemmas carry the tag. In 81 494 of those the old rule already agreed with spaCy, so 104 056 decisions actually move. By part of speech the withheld are 73.0 % NOUN, 22.3 % VERB, 3.1 % ADV, 1.6 % ADJ. PROPN cannot appear: the dictionary is never consulted for a proper noun at all.

A withheld surface is searched as written — the same one-way cost the entity-index veto and the contested-surface rule already carry, and the direction that cannot mislead.

The file is one compact line rather than pretty-printed. A diff is asked which file moved, and one line per sidecar still answers that.

morph[]¶

Five-element arrays, [folded surface, part of speech, case, count, dictionary], sorted. Format 6 and later; format 5 wrote the first four.

A second projection of the same token stream, keyed apart from lemmas. lemmas is keyed on (surface, lemma) and answers what does this reduce to; morph is keyed on (surface, part of speech, case) and answers what is this, and in which case. Two keys rather than one wider one: a contested surface has several candidate lemmas, and folding case into that key would multiply the contest by the cases the surface appeared in, for a reader that never asks both questions together.

It sees the tokens lemmas drops. A token whose lemma is its own surface writes no lemma row, and in this corpus that is most of the invented names — an undeclined name lemmatizes to itself. Those are exactly the tokens whose case nothing else can supply, because Morfeusz has no analysis for them at all.

Not every reading is published, and the rule is in the code. A reading reaches the file when its part of speech is VERB or PROPN, or when its case is Ins or Voc. Each clause has a reader: VERB is the speaker-narration test, which asks set membership over the whole part of speech; PROPN is case on the invented names, whose nominative is also the only subject signal this pipeline publishes, since the dependency parser is excluded; Ins is the frame grammar's role slot, which sits on a common noun (zostać Narratorem); Voc is address.

What is left out is every other case on a NOUN, ADJ, ADV or NUM — 940 094 of the 1 605 330 rows an unrestricted projection would hold (measured 2026-08-12 over a format-6 re-annotation of repozytorium-fabularne-dev at 49fc214; the committed tree is format 4 and carries no morph rows at all until the next run rewrites it). The narrowing is a size decision and the size is measured: publishing every reading costs +81.3 % of the sidecar tree, against +33.0 % for the rule above. A reader that needs a withheld row widens the rule rather than working around its absence.

spans[]¶

A proper noun carries the tagger's lemma, not Morfeusz's. The dictionary is never consulted for a PROPN token, because a dictionary of Polish has no entry for an invented name and its opinion is usually about a different word. What stands instead is spaCy's own lemma: the surface form for an undeclined name, and a guess at a citation form for a declined one. Committed sidecars carry Talxandera → Talxander and Eademossa → Eademoss. A reader that needs the form as written has text.

Labels stay the source pipeline's own. Resolving persName to a Postać is name resolution's job, and nothing in the pipeline knows what the lore contains.

mentions[]¶

Format 10. Candidates, never claims. The ruled gold frame's largest wrong class (46 % of its wrong lines, 123-ner-plan/README.md) is silence on a common-noun expression standing for a lore person, group or creature — orkowie, Piątka elfów, dwór cesarski, rybaka — which no Polish NER scheme tags. With the parser on, every NOUN/PROPN-headed nominal group whose head lemma is in the pipeline's data/role-lexicon.json (590 role lemmas from Morfeusz's personal and animate readings of the attested epithet population, 33 creature lemmas, 48 collective nouns) and that no span already owns is published here with the features a classifier and a consumer need.

Deciding what a candidate refers to is not this file's: the daemon binds it to a session's cast, or a person rules on it as an entity-span row. Measured on 33 texts (99 964 characters): the parser costs +13 % of the run, spans are unchanged, and the layer yields 3.6 candidates per thousand characters. A run passed --no-mentions loads no parser, writes [], and carries no mentions digest in its stamp — a different pipeline, rewriting the tree.

spans is read by GET /entities/{name}/mentions, which answers with the offsets of the name inside each line. Search reads the lemma dictionary and not spans: a query is scored over words, and a mention is located over characters.

The corpus files¶

Four files sit beside the sidecars, rebuilt from them by every run. All four are sorted, tab-separated plain text.

lemmas.tsv¶

folded surface ⇥ lemma ⇥ candidate count. One row per surface the corpus rewrites.

This is the map the search index is built from. Deriving it by walking the tree instead costs about 11.0 s against 0.25 s to read this file — measured when the file was introduced, on a tree of the same order as today's; the 1.1 s of that which was disk is the part that does not move with corpus size. The committed file holds 106 742 rows (measured 2026-08-12 over nerthus.lang/lemmas.tsv in repozytorium-fabularne-dev at 49fc214), projected from 1 654 620 token attestations across 2 207 sidecars. A re-annotation under a different pipeline stamp moves it: the same file rebuilt over a format-6 run comes out at 106 411, which is the size of the disagreement between two model versions rather than a correction. Format 7 moves it again and by more: 105 594, rebuilt over the same corpus at 59e75dc9 — 1 376 rows withheld because the tag settled nothing, 574 new, and 4 613 keeping their surface under a different lemma as the withheld tokens leave the candidate tallies.

A surface whose most-attested candidate is the surface itself gets no row. The absence is the answer: the index finds no rewrite and searches the word as written, which is what the corpus said.

The third column is how many candidates the corpus attested before one was picked. It is triage information for a reader of the diff; nothing branches on it.

Which candidate wins is a property of the corpus, not of the filesystem. Most attestations first, then the lemma's own name for a tie. Nerthus.Core previously took whichever sidecar the directory listing returned first, so two hosts at one commit could build two different indexes. The population that turns on the rule, rebuilt from the sidecars on 2026-08-12 at 49fc214: 3 592 folded surfaces carry more than one candidate lemma, with 273 263 token attestations behind them.

A surface the lore already names is never rewritten, and that gate runs before the tie is broken. The tie rule picks the best of the candidates it is handed and cannot know that all of them are wrong, which is the shape of the failure it removes: the tagger's lemmatizer has no entry for an invented name, invents one, and the frequency rule then settles on it — deterministically, which is worse than arbitrarily, because every host agrees. The published dictionary said nithal ⇥ Nithala, where Nithal is a Lokacja and Nithala is nothing at all.

The gate is an exact match on the folded surface against every ### heading and @alias in nerthus.entities.md, and deliberately nothing looser. Two wider rules were measured and both are wrong. Rebuilding both over the corpus on 2026-08-12 at 49fc214: "the lemma is longer than the surface" would drop 7 067 rows, nearly all correct, because a Polish verb lemma is the infinitive and routinely longer than the form it came from; "the surface is the first word of a multi-word heading" would add 206, nearly all wrong, because place names begin with ordinary adjectives and it would refuse czarna ⇥ czarny on account of Czarna Wieża.

Exact, it drops a few dozen rows out of the 106 742 — 32 by a rebuild of the rule at 49fc214, and that rebuild lands three rows from the shipped file, so read it as the order of magnitude rather than as the pipeline's own count. It has a cost: a handful of those are ordinary plurals — mury, tunele, archiwa — whose singular the lore also carries as a Lokacja, and those words are now searched as written. That is what the index did before this layer existed, and it is the direction that cannot mislead. A missing fold costs a query some recall; a wrong one sends every mention of a place to a block that is not it. An absent entity index vetoes nothing, which is the degradation the rest of the layer follows.

morph.tsv¶

folded surface ⇥ part of speech ⇥ case ⇥ attestations ⇥ dictionary ⇥ name class. One row per reading, so a surface stands in as many rows as the corpus read it ways. Format 6 and later; over a format-4 tree the file is written empty rather than absent, which every reader treats as "no morphology". A four-field row — a format-5 tree — is refused by every reader here rather than read as consent.

The sixth field arrived at format 9 (2026-09-03) and is Morfeusz's name class for the surface — nazwa_pospolita, imię, nazwisko, nazwa_geograficzna, nazwa_własna — as one sorted, |-joined set over every written form the folded key covers, or empty. The pipeline computed it and discarded it through eight formats. It is published as a column and never applied as a rule: a filter built on it was measured to remove as many accepted spans as wrong ones (Ziółka and Kamyk carry surname readings), so it is a feature for a mention classifier, not a verdict. The daemon's reader takes any row of at least five fields and reads the first five, so a format-9 tree reads exactly as a format-8 one did one file over.

Nothing here is vetoed and nothing is contested. The entity-index veto exists because a rewrite can be wrong — it substitutes one string for another and sends a query somewhere. This file substitutes nothing: it reports that a surface was read as a PROPN in the vocative 37 times, so Nithal, whose lemma may never be rewritten, still gets its morphology published. And a surface standing in two cases is not a disagreement to settle — anwarda is genitive in one sentence and accusative in another and both rows are true.

Its two readers in Nerthus.Core are Get-NerthusLangVerbForms, which asks whether a surface is ever a verb, and Get-NerthusLangVocativeForms, which reports how often a form was read as an address against how often it was read at all.

The fifth field is the dictionary's verdict, and it is a published field rather than a lookup for a measured reason. The consumer that needs it — the speaker-narration test — asks whether a surface is ever a verb, and a tagger must not be able to invent a reading SGJP excludes. But the verdict cannot be computed downstream: these keys are folded, and SGJP analyses the written form. Of 47 406 folded surfaces carrying a VERB reading, 36 053 have no analysis at all under their folded key (same measurement, same tree), because powiedział folds to powiedzial. So it is computed per token, on the surface as written, and published.

Where several written forms fold onto one key, admits wins, and that is a known collision. łudzi and ludzi, stań and stan, pomóc and pomoc are each one key here, and the fold strips exactly the diacritics that separate a Polish verb from its noun. 2 095 of the 41 848 keys carrying an admitted VERB reading — 5.0 % — have written forms that disagree, covering 24 188 tokens written in a non-verbal form. On the served speaker worklist the cost is 4 rows of 1 030. It is left open deliberately: demoting every disagreeing key instead puts 152 rows of narration back onto that worklist as noise, which is a 38:1 trade against fixing the 4. Every figure in this section was taken over that same format-6 re-annotation at 49fc214; none of it describes the committed tree, which is still format 4.

index.tsv¶

source ⇥ source_hash ⇥ pipeline ⇥ format_version ⇥ build. One row per annotated source: the three values the skip check compares, and the build that wrote it.

The fifth column arrived at format 7, became the field's ONLY home at format 8, and is not part of the skip check — see build above. It is here because this file is what Get-NerthusLangDigest hashes when the tree is uncommitted or dirty, which is exactly the case a local pipeline run produces. Until 7 the four columns it held were the fields two differently-built hosts agree on, so the file came out byte-identical over a pair of trees whose sidecars differed in 1 288 documents and whose lemmas.tsv differed in 132 rows. A reader older than 7 splits the row and indexes the first four fields, so the extra column costs it nothing.

All three corpus files are a fast path over the sidecars and never an authority on them. They are written from the tree by the run that published them, so a sidecar replaced since is described here as it was — which is why Nerthus.Lang's own skip check reads the file rather than this manifest, and why every reader in Nerthus.Core falls back when they are absent.

names.tsv¶

folded surface ⇥ lemma ⇥ label ⇥ attestation count. One row per multi-word or declined name the corpus attests, built from the spans a format-4 sidecar already carries — no format bump, and a tree written before this file existed grows one on its next publish. Its consumer is name resolution's annotated-lemma stage.

Only name-bearing spans contribute: persName, placeName, orgName, geogName. lemmas.tsv cannot answer this and must not be widened to — its keys are single tokens, while a span carries the whole name (Fortu Eder, Doliny Yss) and its lemma.

The shape differs from the lemma map in two ways. The label travels with the row as a tie-break hint for a reader deciding between two owners of one name — a hint and never a filter, since persName narrowing a lookup to Postać would drop every NPC the tagger read as a place. And the count is the real attestation total, not the lemma map's candidate count: a form said four hundred times and a form said once are different claims.

The entity-index veto and the contested-surface rule are lemmas.tsv's, verbatim: a surface the lore already names gets no row, and a contested surface settles by attestations first, then the lemma's own name for the tie.

A key is a tidied span. Delimiter quotation marks and edge padding are trimmed and internal whitespace collapses when a span becomes a key, so a quoted name ("albatros") or a name broken across a line break folds to one key. Measured 2026-08-12 at 49fc214: the file holds 6 766 rows and none of their keys carries a delimiter quote or edge padding, while 522 distinct name spans in the sidecars would have produced an untidy key had their raw text been used — keys no reader ever looks up.

Offsets¶

A transcript is not a text file, so the pipeline assembles one, and line_offsets records where each line began in that assembly. A span at offset N belongs to the last line whose recorded start does not exceed N.

Two inputs name no line and both answer -1. A lore sidecar carries no line_offsets at all, because a file's own offsets are already addressable. And an offset falling before the first recorded start indexes nothing — a caller's error rather than a position at the top of the document.

Format 1 computed that index and discarded it, which left a reader holding one half of the contract — the offsets — and obliged to rebuild the other half by joining the transcript the same way in another language. nerthus_lang.sidecar.line_for_offset is the resolver on the Python side; the daemon asserts its own against the same fixture.

What gets annotated¶

The eight prose directories — Postaci, Świat gry, Wątki, Organizacje, Bestiariusz, Nerthus, Archiwum, Źródła — plus README.md and REPOZYTORIUM.md, plus every transcript under nerthus.logs/ that holds usable text.

Lokacje/ is not annotated. It is the generated session registry, regenerated byte-identically from the session blocks the pipeline already reads, so annotating it would describe derived rows twice, every lemma a duplicate. Measured 2026-08-12 over tracked content at 49fc214: Lokacje/ is 3.35 MB over 308 files, against 49.55 MB of lore prose — the whole checkout minus the two generated directories, nerthus.lang/ and nerthus.logs/, which are 54.90 MB and 59.01 MB of the 163.46 MB tracked. The registry is a fourteenth of the prose it duplicates, and the reason to skip it is that it is a duplicate rather than that it is large.

nerthus.entities.md, nerthus.contributors.md and nerthus.ledger.md are not annotated, and search indexes all three. A query against the generated index or the ledger matches on surface forms alone. Those files are generated rows rather than prose, and lemmatizing a table of tags buys nothing a substring match does not already answer.

Why a run skips a file¶

Three recorded values must all equal the current ones: format_version, source_hash, and the whole pipeline object. Each guards a different invalidation — a reader change, a lore edit, a model bump — and a model bump invalidates the corpus, so it must not pass quietly.

gazetteer IS among them, and it is the opposite decision from build for the opposite reason. The pattern set decides which spans exist at all, so adding one @alias row to nerthus.entities.md changes the annotation of every document that mentions the aliased name. A stamp that could not say so would leave the tree holding two annotation regimes with nothing recording which sidecar was written under which — the state format_version exists to prevent one file to the side. So an entity-index edit invalidates the corpus and the next run rewrites it. That is expensive, it is visible in the pack, and it is the only honest option: build is out of the comparison because two builds were measured to produce identical annotations, and a changed pattern set provably does not.

build is deliberately not among them, and the omission is a decision rather than an oversight. Since format 7 the annotations no longer depend on which Morfeusz build produced them, so a build inside this comparison would rewrite every sidecar in the tree each time the annotating host changed and alter nothing but its own value — in both directions, forever. What still invalidates a corpus over a Morfeusz change is morfeusz_dict, which is inside pipeline and is compared.

An unreadable sidecar counts as absent and is rewritten. A truncated or hand-edited file should be rebuilt, and a run that stopped to complain about one would leave the corpus half-annotated.

Serialization¶

Sorted keys, , and : separators with no spaces, ensure_ascii off, a trailing newline, written through a temporary file and moved into place. So a diff under nerthus.lang/ means an annotation moved, not that a dictionary iterated differently. Diacritics stay literal because escaping them would triple the file and make review impossible.

.gitattributes in the lore repository holds the whole directory out of end-of-line conversion and out of textual diffing, and puts diff back on the two .tsv files — they are meant to be read in review, and the sidecars are not. ci/publish.sh refuses to publish into a repository where either half of that is missing.

Who reads them¶

Seven readers, not one, enumerated from Nerthus.Core at 3f91ebe by taking every function in daemon/services/Lang.ps1 and asking what calls it. A host running a stale or absent annotation tree loses all of them, which is why the list matters more than any one of its rows.

Six of the seven degrade rather than fail, and the seventh does nothing at all today. That is deliberate and it is stated once here rather than seven times below: a host with no nerthus.lang/ still serves every route, more coarsely.

How Nerthus.Core reads them¶

One surface-form ⇒ lemma map for the whole corpus, read from lemmas.tsv. A missing dictionary sends the daemon to the sidecars — lemmas if the file carries them, tokens if it is format 3 — and a missing directory yields an empty map and the index still builds, on surface forms alone.

The fallback is not decoration. A tree written by an older pipeline has no dictionary in it, and an index that refused to build over such a tree would be a regression dressed as an upgrade.

The map is per corpus rather than per document because a lemma is a property of a word form, and the ambiguous cases are resolved by the run that wrote them. format_version, source_hash and pipeline are not read on this path — the index is keyed on the daemon's read key, so a merged annotation commit invalidates it wholesale.

spans is read one transcript at a time and memoized on the transcript's assembled hash together with the annotation's own size and write time, because either can move without the other: a refetch changes the transcript under an unchanged annotation, and a pipeline run replaces the annotation under an unchanged transcript.

See also¶

• What the language layer is for — what this file is part of, and what it deliberately cannot answer
• The language pipeline — the CI job that writes these files
• Search — the largest of the seven readers above, and what a lemma buys a query
• Work on Nerthus.Lang — running the pipeline locally
• Name resolution — where declining invented names stays