Fetch a source into Repozytorium Dzieł¶

Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: not yet written.

A source has been declared and consented to, and its documents are somewhere on the web. This page turns that into committed markdown a Radny can read in a merge request: the recipe you write, the command you run, the sidecars it writes, and the refusals it writes instead.

What it buys, and what it costs¶

You get one markdown sidecar per document, in a shape that does not depend on what the document was. A consumer reading nocarze.org/content/mundur-bojowy.md and one reading prace-magiczne/riveth/content/zaklinanie.md read the same declaration block, whether the first came off a web page and the second out of a PDF. That is what makes the external search tier a single thing rather than five.

The cost is repository bytes, and it is permanent. Every form keeps its original — the fetched HTML, the fetched PDF, the fetched paste — because a transcription is only evidence when the thing it was made from is still there. A source of 30 PDFs is 30 PDFs in git, forever, and forever means until somebody runs Withdraw a source, which is a history rewrite. Read that page's cost section before you fetch a large source.

No CI minutes. There is no pipeline here. A fetch is a command a Radny runs on their own machine, and the merge request is the act that publishes the work. Nothing on the fleet fetches anything; a host reads what was merged.

Prerequisites¶

• A clone of repozytorium-dziel (or repozytorium-dziel-dev), and push rights on it. Write is Rada.
• A clone of nerthus.core, and lib/ built — see Architecture for the build. The fetch loads the compiled substrate.
• The source's directory exists and its INDEX.md declares @właściciel and @zgoda. A source with no consent line is not fetched, and _template/INDEX.md in the works repository is the shape to copy.

The recipe is data, and it lives with the source¶

There is no list of allowed hosts anywhere in this design. What may be fetched is what a source's own INDEX.md names, and nothing else:

- @przepis: page
- @pobierz: https://nocarze.org/o_nas.html, @forma: page, @plik: o-nas.md
- @pobierz: https://noc.ugu.pl/books/riveth_am_zaklinanie.pdf, @forma: pdf
- @host: googleusercontent.com

@forma overrides @przepis for that row. @plik names the sidecar; without it the name comes from the address. Both the sidecar and the original land inside the source's own directory whatever @plik says.

Two rows may not land on one name, and the second is refused rather than allowed to overwrite the first. A name taken from the address is its last segment, so noc.ugu.pl/ng/39_23.html and nocarze.org/ng/39_23.html both want content/39_23.md — and nocarze's earlier address serves the same paths, which is exactly the pair a real recipe will carry. The refusal names the row that took the name; @plik on the later row resolves it. Nothing is auto-numbered: 39_23.md beside 39_23-2.md would be a guess about which is which, and a guess is what this design refuses to make.

A fetch may reach the addresses its own @pobierz rows name, and a redirect is re-checked at every hop against that same list. @host is how a source widens it, and one form needs that.

The five forms¶

The form decides two things and nothing else: what the fetch expects to come back, and which extractor turns it into markdown. Everything downstream is identical.

primitive and gdoc — the payload is the document¶

Nothing is extracted, so @pewność is 1 and says so. The two differ only in which address is asked for: a Docs link normalises to …/export?format=txt before the fetch, and a bare Pastebin page to /raw/….

A gdoc recipe does not work without an @host line, and this is the reason. The Docs text export always answers 307 to doc-XX-XX-docstext.googleusercontent.com — nine distinct such hostnames appear across the corpus's own attempts — and the walk refuses a host nobody declared. That refusal is the whole history of the mode on the lore corpus: 115 Docs links, 0 ever fetched, 10 of 11 recorded failures exactly this. The mode is not broken and the walk is right; what was missing is a place to say where the export lands. One line supplies it:

- @host: googleusercontent.com

A Docs document must also be readable by anyone with the link. A private one answers 401 and is recorded as a failed fetch, which is the honest outcome — the fix is a share setting, not a scraper.

pdf — plain text, and a scan is refused¶

The extractor reads the document's own text layer. It measures what it got, per document, and writes the number into the sidecar:

- @narzędzie: pdf-text-layer/1
- @pewność: 1
- @uwagi: stron: 1, z tekstem: 1, z obrazem: 0

@pewność is the share of pages that yielded text, bounded by the share of character codes that mapped to Unicode. A document below scrape.pdf_confidence_floor — 0.5 unless a host's config.json says otherwise — writes no sidecar and a .failed.json instead:

{"url": "…/md_m_odrzucenie.pdf", "form": "pdf",
 "error": "pdf: stron: 1, z tekstem: 0, z obrazem: 1; żadna strona nie ma warstwy tekstowej, a 1 ma obraz — dokument jest skanem",
 "original": "originals/md-odrzucenie.pdf"}

No OCR, and that is a standing ruling rather than a gap waiting to be filled. A scanned PDF is recorded and refused; nothing guesses at what the picture says. The refusal names the pages it counted, so the verdict is checkable rather than a shrug.

The original is kept even though the sidecar is refused, and that follows from the same ruling. A scan is the work; only its text layer is missing. Whether somebody's scanned essay should be read by a machine is a decision for the Rada, and when they take it the bytes are already here rather than one re-fetch away.

forum — the pair a moderator can check¶

A thread page becomes one section per post. Posts are found by the anchor the lore corpus itself links to — a session's Logi: line says …&ps=6#post45604265, so the page publishes id="post45604265", and that is the rule rather than a guess about class names. Three optional lines widen it where a forum writes differently:

- @post_klasa: post-wrapper
- @autor_klasa: nick
- @czas_klasa: date

The values are matched as substrings against each element's class and id, case-insensitively.

What the page did not say is recorded, never inferred. A thread whose posts carry no author reads:

- @pewność: 0.5
- @uwagi: postów: 3, bez autora: 3, bez daty: 0

The raw capture sits in originals/ beside it, so a better reader can derive the missing half later without asking the forum again.

forum.margonem.pl wraps no post in anything. Its thread is one table and the posts are separated by <a name="post46905317"></a> alone, so there is no element for @post_klasa to name. The reader falls back to the anchors, which is what the corpus links to anyway, and the post ends where the next one begins. Two lines still matter there:

- @czas_klasa: postid
- @stron: 18

@czas_klasa: postid names the footer cell that carries the timestamp. Without it every post on that forum reads as undated — measured on both granted topics: 9 posts and 30 posts, all with an author, none with a date. The same cell also ends the post: everything from it on is the rating bar and the quote and copy controls, which change on their own.

@stron is the page count you declare. A recipe is a flat list of URLs and cannot page, so this line plus the check against the pager is how a nineteenth page becomes visible. See Capture a source that deletes, below.

page — the main block, and how much was dropped¶

Navigation, sidebars, footers and cookie bars are dropped; what is left is the block carrying the most non-link prose. @pewność is the share of the page's non-furniture text that survived, so 1 means nothing was discarded and a low number is an instruction to look at what went missing:

- @narzędzie: html-main-content/1
- @pewność: 0.039
- @uwagi: blok: div#tresc, znaków: 34 z 864, bloków: 1

A low number is recorded and never a refusal. The text is in the sidecar either way and the capture is beside it.

Capture a source that deletes¶

A fetch rewrites the sidecar when the bytes moved, which is right for a source that keeps its documents and is a data-loss bug for one that removes them. A source that deletes is captured instead: one directory per capture, keyed by the moment it was taken, and nothing ever rewritten.

- @przechwytywanie: dopisywane

That one line makes a source a capture source. The scheduled job in Set up the pipelines captures every source that carries it and skips every source that does not, so which topics are captured is a fact about this repository rather than a list inside a program.

What a capture source holds:

forum-margonem-515526/
  INDEX.md
  przechwyty/index.jsonl            every attempt, including the failures
  przechwyty/2026-08-11T193133Z/    one capture: page-NN.html, page-NN.headers, page-NN.md, SHA256SUMS
  derived/log-zmian.jsonl           the events the daemon reads, rebuilt from every capture

Four things to know about that layout:

• SHA256SUMS names its files relative to the capture directory, and is verified from inside it. An absolute path verifies only on the machine that wrote it.
• A capture never lands under content/. The external search tier indexes every content/**/*.md, and a year of daily captures of one topic would put 365 near-identical documents into search results.
• index.jsonl carries every attempt, with its timestamp, its outcome, the post ids it saw and a digest of their content. A day with no row is a day nobody looked, and that is the difference the record exists to keep — a quiet week and a job that never ran look identical everywhere else.
• The digest reads the posts, not the page. A forum page carries a day-and-night banner, per-author reputation counters and avatar paths, all of which move without a post moving: measured over 17 hours on topic 515526, the bytes differed in three places and no post did.
• The digest cannot see the author. It covers the post id, its timestamp, its Edytowany lines and its body — not the nick. A poster renamed between captures produces an identical content_digest while the transcription beside it, which writes ## <id> — <author> — <timestamp>, changes. Proven by mutation, with two controls in the same instrument that do move it. Nothing branches on the digest today; anything that starts to — a skip-if-unchanged write, a "nothing changed" report — has to say the author is outside it.

What a sidecar looks like¶

One shape, five forms. Every key is present on every sidecar; a missing one means somebody edited the file by hand.

- @ścieżka: nocarze.org/content/mundur-bojowy.md
- @forma: page
- @url: https://nocarze.org/o_nas.html
- @pobrano: 2026-08-10T18:22:31.4471234Z
- @suma_kontrolna: sha256:1ca4517b302f7eb9…
- @oryginał: originals/mundur-bojowy.html
- @wywiedzione_z: sha256:1ca4517b302f7eb9…
- @autorytet: external
- @diegeza: mixed
- @narzędzie: html-main-content/1
- @pewność: 1
- @uwagi: blok: td.tresc, znaków: 254 z 254, bloków: 8

# Mundur bojowy

Mundur bojowy nosi się w Werbinie i w Ithanie…

@wywiedzione_z is the sha256 of the file @oryginał names, so the transcription and its evidence are a checkable pair — the discipline the regulations corpus already runs on, applied to every form rather than to the ones that obviously need it.

The declaration block is not indexed. A reader searching the corpus gets the document, never a fetch record.

Steps¶

1. Write the recipe into the source's INDEX.md, following the shape above. _template/INDEX.md carries the full grammar.

2. Preview. The default writes nothing and lists what each row would fetch and where it would land:

   ./vm/Invoke-NerthusSourceFetch.ps1 -Works ~/repozytorium-dziel-dev
   

   Add -Source nocarze.org to run one source instead of every declared one.

3. Fetch.

   ./vm/Invoke-NerthusSourceFetch.ps1 -Works ~/repozytorium-dziel-dev -Source nocarze.org -Apply
   

4. Read the diff, then open the merge request. The merge is what publishes the work, so the review is a person reading the sidecars — not a check the tool ran.

How you know it worked¶

The last line reports each outcome separately:

sources 1, rows 6: fetched 5, unchanged 0, refused 1

Read all four. A run that fetched nothing and refused nothing did nothing, and a total alone would hide one number growing while another shrank.

Then: every content/*.md has a file in originals/ beside it, and git status shows both. Re-run with -Apply and every row reports Unchanged — a second run rewrites nothing at all, @pobrano included, so a fetch that changes files is a fetch where something upstream moved.

When it did not¶

A refusal writes content/<name>.failed.json and never a sidecar, so a refused document cannot reach the search tier by accident. The marker is committed rather than cached: which of a source's documents could not be fetched is part of that source's record.

A name is held by the row that claimed it whether or not that row's fetch succeeded. So a collision refusal can name a row that wrote nothing, and the resolution is still @plik — releasing the claim on a failure would make the second row's outcome depend on how the first host answered that day.

Adding a form¶

You cannot, from a recipe. The five are a closed table in daemon/services/Scrape.ps1, and a sixth is a code change with its own tests — the content gate runs before every extractor because the frame dispatches through that table, and a form added any other way would bypass it.

Related¶

• Withdraw a source from Repozytorium Dzieł — honouring a withdrawal of consent
• Ecosystem overview — where the works repository sits
• Search — the external tier the sidecars are read through
• Configuration — corpora.dziel, scrape, content_gate