Manage notifications¶

Nerthus.Core (until cutover). This page describes the frozen system that runs today and is deleted at cutover. Replaced by: not yet written.

Every fleet notice — provisioning outcomes, health transitions, and takeover steps — posts to the one IT tech channel kanal-wewnetrzny-it (the committed IT role's @discord, the name pair kanal-wewnetrzny-it, it). This page owns the wiring and the event catalog. The notices live in a Worker and the daemon rather than CI because health runs continuously; Fleet owns that budget doctrine.

The one delivery mechanism¶

There is no webhook. The daemon, the health Worker, and fleet CI all deliver the same way: a bot POST to the channel resolved by name, the mechanism the daemon has always used for lore. Each sender holds the fleet bot's token, resolves channel, category to a channel id with GET /guilds/{id}/channels, then POSTs to /channels/{id}/messages. All three send the required DiscordBot (…) User-Agent; Discord's edge rejects a guild request without it as a 403 carrying code 40333, while account-level calls still pass, so the failure looks misleadingly like an outage.

Sender	Job or process	Config it reads
daemon	the primary host	`DISCORD_BOT_TOKEN` (from `secrets/<town>.discord.age`, decrypted by the agent and rendered into the compose env on the primary only) + `DISCORD_GUILD_ID` (from `discord_guild_id` in `fleet.settings.yml`), `integrations.discord.enabled`
health Worker	`fleet-health`	`DISCORD_BOT_TOKEN` (wrangler secret), `DISCORD_GUILD_ID` + `FLEET_ALERT_CHANNEL` (wrangler vars)
fleet CI	`fleet:notify`	`DISCORD_BOT_TOKEN` + `DISCORD_GUILD_ID` (CI variables)

The bot must be in the guild with View Channel + Send Messages, invite permissions 3072. Without View Channel every resolve returns "no channel" and nothing posts. One secret value is shared by all three holders; rotating it touches each (Rotate secrets).

What provisioning reports¶

Fleet CI's fleet:notify posts one message per provision push, ending on what is still pending rather than on success:

Flota <env>: pipeline #<iid>, MR !<n> (autor: <user>) — the header.
Host <town>: Skonfigurowany - tunnel: utworzono, … — one line per host that changed.
Flota <env>: Oczekiwanie na uruchomienie hostów (<town>, <town>, ...) przez właścicieli (<owner>, <owner>, ...).
Flota <env>: Bez zmian — hosts.yaml zgodny z rejestrem — a no-op run.
Błąd: Brak fleet-changes.json - aktualizacja Floty (<env>) nie powiodła się — provision failed early.

The Zaktualizowany confirmation that a provisioned host is actually running comes later, from the sweep.

What a converge reports¶

Each daemon posts one line per repository each time its sync tick fast-forwards onto a commit it did not already hold — the lore checkout and every corpus. Host <town>: <repozytorium> (<key>) na <sha>, followed by the author or authors (%an/%ae, --no-merges) and a count of commits and files. It answers "is the fleet serving what I merged yet", which the provisioning notice cannot: that one fires when CI finishes, minutes to a quarter of an hour earlier.

Only a fast-forward posts. A tick that pushed is a host reporting its own write, a tick that found nothing has no news, and a sha already announced on that host is silent. The knobs — including the channel and the off switch — are integrations.discord.converge_announce.

Primary only, today. The agent renders DISCORD_BOT_TOKEN for ROLE = primary alone, so a replica's announce lands in its delivery log as SKIP/no_token. Giving replicas a token is what would make the per-host reading complete, and that is a secrets decision.

What the health sweep reports¶

The Worker speaks only when a host's state changes, never on an unchanged pass, so a fleet of sleeping laptops stays quiet.

Alert	Fires when
`Host <town>: Brak odpowiedzi - <error>`	a host stops answering `/health`
`Host <town>: Znów odpowiada (OK)`	it answers again
`Host <town>: Obraz Docker nieaktualny - <obs> (wymagany: <pin>)`	it runs a digest other than the pin
`Host <town>: Obraz Docker aktualny (OK)`	back on the pinned image
`Host <town>: Schemat nieaktualny - <s> (wymagany: <exp>)`	its on-disk schema differs from expected
`Host <town>: Schemat aktualny (OK)`	schema matches again
`Host <town>: Brak aktualizacji zmian od <n> minut`	sync went stale (over 60 min)
`Host <town>: Pobiera aktualne zmiany (OK)`	sync is fresh again
`Host <town>: Zaktualizowany - obraz <image>`	first sweep that sees a host on a newly-pinned image; closes provision's deferred confirmation
`Flota <env>: Hosty nieaktualne od 24 godzin - <town>, <town>, ...`	a pinned image still not taken up after 24 h
`Flota <env>: Raport dobowy - dostępność: <up>/<n> hostów, obraz Docker: <image>`	once a day; its absence is the signal the Worker itself stopped
`Flota <env>: Monitor aktywny - Worker w trybie czuwania (0 operacji KV)`	the standby gate saw fresh monitor state and the Worker stood down
`Flota <env>: Monitor nie odpowiada - Worker wznawia sweep`	the monitor went dark; the Worker's very next sweep took over

What automatic takeover reports¶

The takeover epoch — the fleet's promotion lineage count — is tracked internally and no longer appears in the message text. The procedure behind these is Run a failover.

Alert	Step
`Host APEX <primary>: Brak odpowiedzi od ~<mins> minut - szukanie nowego hosta APEX (kandydat: <town>)`	armed after about `after_minutes` of confirmed darkness (default 30)
`Zmiana hosta APEX: <primary> → <town> (MR !<iid>, auto-merge)`	the role-flip MR is open
`Nowy host APEX: <town>`	the MR merged and provision enacted it
`Ręczna zmiana hosta APEX: <a> → <b> (automat wstrzymany na <h> h)`	a human flipped the roles instead
`Błąd: Zmiana hosta APEX zablokowana - MR !<iid> nie został scalony, sprawdź pipeline`	the MR sat unmerged past 30 wall-clock minutes
`Błąd: Brak kandydata na hosta APEX - <why>`	the primary is down and no replica qualifies
`Nie udało się otworzyć MR przejęcia …: <error>`	the actuation token or API failed
`Przejęto istniejący MR zmiany hosta APEX - MR !<iid> (drugi proponent)`	the pre-flight found another proposer's open takeover MR and adopted it instead of racing a second one
`Zmiana hosta APEX już scalona - MR !<iid> (automat wstrzymany do końca cooldownu)`	the pre-flight found a takeover already merged inside the cooldown and honoured it

The daemons contribute their own mesh lines through the same channel (primary only — replicas hold no bot token): peer-dark edges and recoveries (… (peer watch)), a clock skew alert, the split-brain finding (report-only, never acted on), and the Discord-dispatch streak transitions (Dyspozycja Discord: …).

Split-brain posts on edges: one message per finding, not one per watch tick. The daemon posts Błąd: <finding> (split-brain - nic nie zmieniam, role zmienia tylko git) on the tick a finding first appears. It posts OK: rozbieżność primary ustąpiła, tylko jeden host twierdzi, że jest primary (split-brain) on the tick the last finding drops out. A finding that merely persists costs no further message. Who claims primary reads out of GET /fleet/peers — self.role and each peers[].card.role, cached from the last probe rather than fetched live. The operational stream takes every finding on every tick. With no contributor store loaded, or no @discord on its IT role block, the sender falls back to the kanal-wewnetrzny-it, it name pair.

A Błąd: notice is the fleet asking for a human; everything else is informational.

Verifying delivery¶

The bot is write-only (no Read Message History), so it cannot read its own posts back. Confirm by eye in the channel, or read the last sweep at the Worker's plain GET / (KV fleet-state.json), which records every host's reachability, skew, and takeover state without posting. A fleet:notify run that cannot resolve the channel throws and fails the job; a Worker that cannot resolve silently no-ops, because a sweep must never fail on delivery.