Records the deployed state: fallback expressions, directory mounts (which proved
themselves on the very next rollout, where a SIGHUP reload was genuinely enough),
and alertmanager now scraped so delivery failures are visible. Also notes the
README correction — delivery had not been muted since gitops#51, and docs saying
otherwise would have made a missing alert look expected.
Left open deliberately: TrivyScanStale has the same missing-series gap but no
natural equivalent to kube_cronjob_created, so it needs a decision rather than a
reflex; and phase A/B of the restore drill still needs a throwaway environment.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Alert rules deployed and verified on CFGMON. Verification surfaced two variants
of the same failure class: a missing time series silences an alert instead of
firing it (fixed with a created-time fallback aggregated via max by, plus an
absent() alert for a vanished CronJob), and a SIGHUP reload that reported success
while serving the old file from a stale inode.
The second one matters most: that trap was already documented in detail, with the
right command and a check, and it still bit — so it was removed structurally
(directory mounts) rather than documented harder.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Monthly restore-drill CronJob (verified before commit) plus the finding that
mattered more: there was no backup alerting at all, so a failed nightly job
would have gone unnoticed. Added BackupJobFailed/BackupNotRunning/
RestoreDrillStale; the last one alerts on the absence of the check itself.
Alert rules still need deploying on CFGMON.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Standing exception to ADR-0001 (everything is push-mirrored to Gitea). The
handbook necessarily maps the infrastructure, the backup locations and where the
keys are kept; mirroring it onto the internet-facing host that is itself one of
the covered failure cases would hand a post-compromise attacker their next step.
Confidentiality over availability, with a local clone closing the availability
gap. Records the rejected alternatives so the question does not reopen.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Not mirroring it is deliberate: the handbook maps the infrastructure, backup
locations and where the keys live, so putting it on the internet-facing Gitea
hands an attacker the roadmap once the stack is compromised. My earlier
recommendation only weighed availability and was wrong. Local clone covers the
availability gap. Flagged that this is a standing exception to ADR-0001 and
would warrant its own ADR.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The databases are no longer an assumption: notfall.sh stage 3 restored all
three Borg repos into a throwaway postgres inside the pod (synapse 31908 rows,
MAS 16085, authentik 325149, wiki 251), isolated from production and repeatable.
Procedure and tool now live in git.lab/axion1337.chat/notfallhandbuch so an
emergency needs one clone; this repo keeps a pointer. Still open: phase A/B on
an empty host, Synapse media, a repeat cadence, and mirroring that new repo off
git.lab.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Derived from the running system: prerequisites (age key from the vault first),
the three Borg repos with their archive layout, bootstrap order (host/K3s, the
two manual secrets, Flux, kustomization dependencies), data restore and
verification. Lives in git rather than Wiki.js on purpose — the wiki runs on the
cluster being restored.
Two previously undocumented findings: Flux pulls from Gitea rather than git.lab,
so a simultaneous loss of rohana requires repointing gotk-sync first; and
consumers must be scaled down before pg_restore or their startup schema collides.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sorb's call: Gitea on rohana is a push mirror of the canonical git.lab, so a
nightly gitea dump would back up a copy — effort not justified, cron stays off.
Documented the one non-mirror asset for the record: the container registry holds
four images the cluster pulls (incl. threadnet-web and the backup image itself),
which is rebuild time rather than data loss and is covered by #0022/#0033.
For #0030, sorb confirms the age key is also in the password vault, dissolving
the circular dependency found earlier. Noted that the vault is now part of the
restore path and must lead the procedure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cluster-side backups are healthier than assumed: three nightly Borg jobs to a
Hetzner Storage Box, all completing with plausible volumes and working prune
(synapse 199MB/247 files, authentik ~150MB, wikijs 223kB DB-only). No silent
failures.
Critical finding for #0030: the Borg passphrase and SSH key needed to READ those
backups are SOPS-encrypted under a single age key that exists only in the cluster
being backed up and on one laptop — no documented cold copy. Losing both makes all
three repos permanently unreadable. Cold escrow must precede any restore drill.
For #0010 this shrinks the work: the Storage Box + Borg pattern already exists and
is proven, so Gitea needs only its own repo there. The disabled cron (no backups
since 2026-07-30) remains separately urgent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Gate 4, slice 4: 26 open management issues imported from live git.lab
(read-only, descriptions included as authorized; GitLab iid = file id,
labels/milestone/priority/due/host/area mapped into frontmatter, the
import aborts instead of inventing a missing milestone or priority).
Two new issues close the F-004 gap where work was really still open
(0033 OVERMIND-01, 0034 CFGMON-11 incl. the plaintext npm-token
rotation); CFGMON-12/13 already route to verified git.lab issues,
MATRIX-05 is done and needs none (agreed with sorb). The three wiki
task blocks now reference their issues, roadmap.md hands all counts to
the generated STATUS.md and states M1-M5 per ADR-0010 (closing F-001
in the canonical prose), pruefe_prosa joins the CI validate job, and
the import protocol under docs/sources/migration/ records every
intervention into imported text.
Verified: validate 0/0 over 29 issue files, gen_status --check current
(distribution line M1 9 - M2 17 - M4 2 plus per-issue milestone and
priority), pruefe_prosa 0 errors with clones and 0 errors/15 unchecked
citations in offline CI mode, drift check green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Thore CimbalandClaude Opus 4.8