Search¶

Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: not yet written.

Two surfaces in this repository had no way to be searched. The lore is two and a half million words of Markdown, reachable a file at a time through Lore files and searchable by nothing. And the speech layer's ?text= filter on Logs & audit reads every candidate transcript off disk to run a substring match over it, so it costs the corpus rather than the match and cannot rank what it finds.

These routes answer both from an inverted index: BM25 ranking, Polish lemmas where the corpus has been annotated, snippets around the hit, and an optional date range.

Routes¶

client/Cmdlets-Search.ps1 wraps all four. The daemon-side Find-NerthusSearchService and friends are internal to the handler and are not a client surface.

Find-NerthusSearchResult returns the whole envelope rather than unwrapping to .items the way Find-NerthusEntity does, and the asymmetry is on purpose: on this route an empty result has three distinct causes — nothing matched, the host builds no index, or the caller may not read the tier they asked for — and only Searched, Skipped and Serving tell them apart. A cmdlet that returned the items alone would collapse all three into the same empty list.

The rebuild is Invoke- rather than Update- because the module exports an approved Verb-Nerthus* surface (Nerthus.Core.psm1) and Update is not in it. A function named Update-Nerthus* loads and is then silently never exported.

† The declared capability is a floor, not the whole check. See What a caller may search.

GET /search is replica-safe and poolable. POST /search/index/rebuild refreshes derived state over a committed tree and writes no lore, so it carries no write flag — the same trade /logs/index/rebuild and /name-index/rebuild make.

Paths are relative to /v1/api; the cross-cutting contract is on the API reference index.

What a caller may search¶

The corpus sits behind four different capabilities, and the route respects all of them rather than picking one.

† Who actually holds it is whatever the repository's roles.json says. A role's capabilities come from the committed data-tables/roles.json, which replaces the built-in bundles rather than merging over them, and that file is seeded only when it is absent. So a repository adopted after this capability existed is correct with no edit, and one adopted before keeps the table it was seeded with until the Rada adds the key — until then only admin.all reaches the tier. Data tables owns the seed-only-when-absent rule, and Capabilities the two-step the edit actually takes.

speech is scoped by the channel gate the speech layer owns: Lokalny or no channel at all, and never a line carrying an addressee.

external walks Repozytorium Dzieł, a second working tree named by corpora.dziel (Configuration): every <source>/content/**/*.md sidecar, plus the dated <source>/RRRR-MM.md files a hand-placed source drops at its own root. A source directory with no INDEX.md is not walked at all, and neither is _template/ or any INDEX.md — a declaration names an owner and a consent, and search results are the wrong place for either.

A host with no works tree answers the tier empty rather than failing. No configured root, a configured root that is not there, a tree that declares no source: each reports the tier searched and nothing found, which is a different fact from being refused and reads differently to whoever asked.

A Gracz holds entity.read alone. They can open lore prose through /files/content, so gating lore search behind another capability would take away something they have; and answering their search from the transcripts or from the session blocks would give them what the role withholds. A session write-up carries what the narrator told the table and what was withheld from it, so being present at an evening is not a claim on its record — the same rule /sessions enforces, applied to the index that would otherwise route around it.

So the route declares entity.read to authenticate the request, and the handler intersects the requested scope with what the caller actually holds. Every answer says which tiers it searched:

{
  "Query": "karczma",
  "Scope": "all",
  "Searched": ["lore"],
  "Skipped": ["session", "speech", "external"],
  "Count": 3,
  "Total": 47,
  "Offset": 0,
  "Limit": 20,
  "Items": [ ... ]
}

Skipped is not decoration. A shorter result list with no explanation is how a search surface teaches people it is broken.

Count is this page and Total is the population. A caller reading the first as the second reads a page size as a fact about the corpus, which is the one thing a search must never say. Total counts one row per (tier, source) — a file indexed in two tiers answers twice on purpose, because a lore section and a session block are different answers about it — and it is exactly what ?offset= walks. A page past the end is an empty page with the same Total, not a refusal and not an empty corpus.

Two consequences worth stating plainly:

• An .own grant never admits a tier. log.read.own scopes an answer to the caller's own Gracz and Postacie, and an index has no such scoping to apply — every posting and every term statistic is corpus-wide. Admitting it would hand over the whole archive under a capability that promises the opposite.
• Each tier is a separate index, not one index with a visibility field. Result counts, corpus term statistics, and the bare fact that a rare word matches something are all disclosures that survive redacting a snippet, and none of them is visible to a route-level capability check. Separate posting spaces keep the term statistics inside the tier that owns them.

A caller holding neither capability for the scope it asked for gets 403 SearchScopeForbidden, with the tiers it was refused.

Query parameters¶

Every filter here refuses a value it cannot honour, and the reason is one sentence. The scorer compares kind by string equality and dates by ordinal string comparison, so an unrecognised value does not merely return nothing — a malformed bound admits and excludes documents arbitrarily. Either way a caller handed an empty page reads it as the corpus has none, and that is the answer a research tool must never fabricate.

GET /v1/api/search?q=karczma%20Karka-han&scope=lore&limit=5
GET /v1/api/search?q=karczma&limit=20&offset=20
GET /v1/api/search?q=danina&from=2019-01-01&to=2021-12-31
GET /v1/api/search?q=zakl%C4%99cie&scope=external&diegesis=in-game

An unknown diegesis is refused rather than answered empty. A caller who typed outofgame and was handed nothing would read it as "the corpus holds none", which is the one answer this filter must never fabricate.

asOf and the from/to pair answer different questions — "what had been established by then" against "what was said during" — so sending both is 400 SearchBoundConflict rather than a precedence rule that would be wrong for one of them.

A dated range excludes undated documents. Nothing records when an undated claim was true, so no range can honestly include one — a filter that admitted them would report a certainty the corpus does not have.

Only the speech tier is dated. A transcript takes the date of the session whose @Logi names it, the earliest where several do. Lore files carry no date: entity and location prose is undated, and taking a date from git would record a reflow commit as a fact changing. So a bounded query answers over the transcripts, and the lore tier honestly answers nothing.

What a hit looks like¶

{
  "kind": "speech",
  "source": "pastebin.com_raw_8Gw4nT2p_A1B2C3D4",
  "title": "Karendar",
  "header": "2025-09-23, Radnerak podejmuje próbę dyplomacji wobec Ithan, Karendar",
  "line": 109,
  "date": "2025-09-23",
  "authority": "transcript",
  "diegesis": "in-game",
  "score": 6.2214,
  "matched": ["karczma"],
  "snippet": "…skręcili do karczmy pod Złotą Wywerną, gdzie czekał już…"
}

source is a repository-relative file for a lore hit, the archive sidecar for a speech one, and a path relative to the works tree for an external one; line is the ordinal within that transcript and is null for a whole-file document. matched names which of the query's terms the document actually carried, so a caller can say why a row is there. score is BM25 and is comparable only within one answer.

header is the evening the hit belongs to, and it is null on a lore hit — for the same reason date is. Entity and location prose belongs to no session, so the field is absent rather than guessed.

On a session hit it is the dated heading the block sits under, taken from the trail in title. On a speech hit it is resolved through @Logi: the session that cites this transcript. Where several cite one transcript — a log re-used across two evenings — the earliest-dated session wins, which is the same rule that picks the hit's date. That is deliberate rather than incidental: two rules would put a 2022 date beside a 2024 header on one row, and each field would be defensible on its own while the pair was nonsense. A transcript cannot predate the session that first references it.

A transcript no session names has no header, and the field is null there too.

The index holds offsets rather than text — the same trade the speech index makes — so a snippet is read back from the source at answer time. A source that moved since the index was built yields an empty snippet rather than a wrong one.

Provenance: two axes on every hit¶

authority says who vouches for a document; diegesis says which world it speaks from. Both are assigned when the index is built, where tier membership already is, and a hit reports what the postings were built with rather than re-deriving it from the path.

The lore tree's values are constant per root: every root is rada, and every directory root is in-game — the one root that needed a mixed verdict, the public website's Nerthus/, left for a repository of its own in August 2026. Among the root-level files, nerthus.entities.md is in-game; nerthus.contributors.md, nerthus.ledger.md, README.md and REPOZYTORIUM.md are out-of-game; the two data tables nerthus.maps.md and nerthus.pu-sessions.md are undeclared and carry the mixed fallback. A works document takes the two values its source's INDEX.md declares, and a per-file override line there wins over them.

Two consequences worth stating plainly:

• A ?diegesis= bound excludes a document that carries none, on the same rule the date bound follows: an unlabelled row admitted silently would report a certainty the corpus does not have. An index built before provenance existed carries none for every document, so a bounded query over a stale cache answers nothing until the tier rebuilds.
• Authority governs what the machine may assert, never what counts as evidence. A name used in a player's own work is real usage and is counted; what the cap forbids is asserting an identity from it. A worklist row resting on evidence below rada or transcript never exceeds średnia and never reaches the exact tier — enforced once, at the conflict veto in Name resolution's normalization pass, rather than in each surface that mints rows.

One document answers once, and the tiers take turns¶

A lore file is indexed as sections, not whole, so title is the trail a hit sits under: the directory path, the file, then the headings — Świat gry > Werbin > Miasto > Polityka > Polityka. The path is part of it because the tree files a document under its subject, and a page about a town often never writes the town's name in its body. Two rules shape the page that comes back, and both were measured rather than assumed:

• One row per document. A file answers with its best section and a transcript with its best line. Without it a page of five results was routinely five sections of one file, which answers a question nobody asked: the caller wants to know which documents are about this, not which paragraph of one of them.
• A dated section is a session record, not lore. A character file is mostly re-pasted session blocks — the same paragraph appears in every participant's file — so a term used once at the table would otherwise answer with five dossiers instead of the page describing what the term is. The two kinds are also two tiers behind two capabilities, so the distinction decides both what a hit answers and who is answered at all.
• The kinds alternate, prose first. Scores from the two indexes are not comparable in practice — a five-word utterance carries a term at enormous density and a three-hundred-word section does not, and BM25 rewards the short document. Merged by raw score, a query for a town name returned five people saying its name and no lore at all. Ask for one kind with ?scope=.

The generated entity index and the ledger echo are not searched. Chunked, they are thousands of tiny blocks each carrying a name and nothing else, and they took every slot. /entities/search already owns "which entity does this name reach", which is the question they answer.

Polish inflection¶

Where the corpus has been annotated by Nerthus.Lang, the index carries a lemma beside every surface form, and a query is reduced the same way. The lemma comes from the committed annotation sidecars, read once per index build. So karczmy finds a document that says karczmie, which plain substring matching cannot do in a language where most words are inflected most of the time.

Two limits are deliberate:

• The dictionary never rewrites an invented name. A dictionary of Polish has no entry for Karka-han or Solmyra, and where it has an opinion anyway it is usually about a different word, so Morfeusz is not consulted for a proper noun. The tagger's own lemma stands instead, and a surface the lore already names is vetoed rather than rewritten. Declining invented names stays with the name resolution engine, which is built for exactly the words a dictionary lacks.
• Without the annotation tree the index still builds, on surface forms alone. That is what the daemon could do before, not a failure.
• Three indexed files carry no lemmas. The pipeline annotates prose, so nerthus.entities.md, nerthus.contributors.md and nerthus.ledger.md are indexed but never annotated. A query hitting one of them matches its surface forms.

Diacritics are folded on both sides, so Zażółć and zazolc are one term. The corpus is inconsistent about them in nicks and headings, and this is the one place that is reconciled.

Index state and rebuilds¶

GET /search/index reports what this host holds, per tier, without ever building one — a route asked "is the index current" must not spend a minute deciding.

{ "Tiers": [
  { "tier": "lore",    "capability": "entity.read",  "state": "fresh",   "documents": 1365, "cached": true },
  { "tier": "session", "capability": "session.read", "state": "fresh",   "documents": 4820, "cached": true },
  { "tier": "speech",  "capability": "log.read",     "state": "on-disk", "documents": null, "cached": true },
  { "tier": "external","capability": "external.read","state": "absent",  "documents": null, "cached": false }
] }

state is fresh when the index is loaded in this process, on-disk when a current one is cached but not yet loaded, absent when the next search will build one, and disabled when this host does not build that tier.

A host chooses which tiers it builds¶

An index is memory this host holds for as long as it serves, so building one is a decision about the machine rather than about the lore — the same posture the reader pool takes. server.search_index names the tiers: lore, session, speech, all, or a comma-joined list. Naming lore enables session with it — the two are one walk of one tree split by kind, so a host that pays for the lore index has already paid for the session one. Empty means none, which is what a host gets unless it asks.

The two are separate because they cost differently for different value. Measured on the reference corpus after chunking, the lore index holds 123 MB of managed objects against a fleet memory limit of 3072 MB — and a cold load of the lore artifact allocates roughly twice its resident size on the way in, which is the figure that matters on a host near the cap. And it is the lore tier that closes a gap — two and a half million words nothing could search — where the speech tier improves on a ?text= filter that already answers. A host with room for one should say lore.

A host that has not opted in still serves these routes. /search answers 503 SearchIndexDisabled for a tier it does not build, which is a different fact from an empty result and reads differently to whoever asked. It will also still serve an index a peer handed it through /derived — the setting governs deriving an index, not holding one.

An index is keyed on the identity the rest of the daemon already mints: the read key, the annotation digest — a pipeline run writes nerthus.lang/ without touching a commit — and, for the speech tier, the archive digest, because a fetch rewrites sidecars while every session block stays byte-identical. Beside those sits the index format's own version, and nothing else about the daemon that built the file. An index whose stamp does not match the current one is never served; it is rebuilt.

POST /search/index/rebuild forces that, and takes ?async=true for the 202 job envelope. It is also how a host adopts a change in which lines a tier admits: such a change moves no stamp, so an index built under the earlier rule reads as current by its own identity until the format version moves.

Copying an index instead of building one¶

The indexes are published as derived artifacts, so a host can take a copy from a peer that already built one rather than deriving it again:

They are three kinds rather than one file with a tier field, and that is the security argument rather than a layout preference. The peer token carries derive.read; separate artifacts are what let a host be allowed to copy the lore index without thereby being allowed to copy the transcript index — and the same argument holds for the session index, which sits behind session.read. One file would make that a filtering question, and the import path does not filter — it checks that a payload's own stamp equals the stamp this host holds, and refuses it otherwise.

The external tier has no derived kind, deliberately. Handing an index between hosts saves the cost of building one, and the works corpus is three sources rather than two and a half million words — building it is cheap enough that the artifact would not earn the surface it adds. A host that wants the tier enables it and builds its own.

Refusals¶

What this host could not answer¶

GET /search/zero-results lists the queries that came back empty, newest first, with ?limit= up to 200. Which searches find nothing is the cheapest signal about where the lore has holes.

It is the one read on this surface that is not a function of the committed tree, and it says so: the payload carries Local: true, the route carries no Replica flag and no ETag, and two hosts legitimately disagree about what was asked of them.

Three things are deliberately absent from a row. The caller — a log naming who searched for what is surveillance of narrators, not hygiene. A capability refusal — a Gracz asking for transcripts is told the tier was skipped, which says nothing about the corpus. And an unbuilt tier, for the same reason: unbuilt is not empty.

See also¶

• Lore files — reading the file a hit points at
• Logs & audit — the transcript a speech hit came from
• Name resolution — which entity a name reaches, a different question from which documents mention it
• Fleet mesh — the derived-artifact wire these indexes travel on