Monthly restore-drill CronJob (verified before commit) plus the finding that
mattered more: there was no backup alerting at all, so a failed nightly job
would have gone unnoticed. Added BackupJobFailed/BackupNotRunning/
RestoreDrillStale; the last one alerts on the absence of the check itself.
Alert rules still need deploying on CFGMON.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Standing exception to ADR-0001 (everything is push-mirrored to Gitea). The
handbook necessarily maps the infrastructure, the backup locations and where the
keys are kept; mirroring it onto the internet-facing host that is itself one of
the covered failure cases would hand a post-compromise attacker their next step.
Confidentiality over availability, with a local clone closing the availability
gap. Records the rejected alternatives so the question does not reopen.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Not mirroring it is deliberate: the handbook maps the infrastructure, backup
locations and where the keys live, so putting it on the internet-facing Gitea
hands an attacker the roadmap once the stack is compromised. My earlier
recommendation only weighed availability and was wrong. Local clone covers the
availability gap. Flagged that this is a standing exception to ADR-0001 and
would warrant its own ADR.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The databases are no longer an assumption: notfall.sh stage 3 restored all
three Borg repos into a throwaway postgres inside the pod (synapse 31908 rows,
MAS 16085, authentik 325149, wiki 251), isolated from production and repeatable.
Procedure and tool now live in git.lab/axion1337.chat/notfallhandbuch so an
emergency needs one clone; this repo keeps a pointer. Still open: phase A/B on
an empty host, Synapse media, a repeat cadence, and mirroring that new repo off
git.lab.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Derived from the running system: prerequisites (age key from the vault first),
the three Borg repos with their archive layout, bootstrap order (host/K3s, the
two manual secrets, Flux, kustomization dependencies), data restore and
verification. Lives in git rather than Wiki.js on purpose — the wiki runs on the
cluster being restored.
Two previously undocumented findings: Flux pulls from Gitea rather than git.lab,
so a simultaneous loss of rohana requires repointing gotk-sync first; and
consumers must be scaled down before pg_restore or their startup schema collides.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sorb's call: Gitea on rohana is a push mirror of the canonical git.lab, so a
nightly gitea dump would back up a copy — effort not justified, cron stays off.
Documented the one non-mirror asset for the record: the container registry holds
four images the cluster pulls (incl. threadnet-web and the backup image itself),
which is rebuild time rather than data loss and is covered by #0022/#0033.
For #0030, sorb confirms the age key is also in the password vault, dissolving
the circular dependency found earlier. Noted that the vault is now part of the
restore path and must lead the procedure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cluster-side backups are healthier than assumed: three nightly Borg jobs to a
Hetzner Storage Box, all completing with plausible volumes and working prune
(synapse 199MB/247 files, authentik ~150MB, wikijs 223kB DB-only). No silent
failures.
Critical finding for #0030: the Borg passphrase and SSH key needed to READ those
backups are SOPS-encrypted under a single age key that exists only in the cluster
being backed up and on one laptop — no documented cold copy. Losing both makes all
three repos permanently unreadable. Cold escrow must precede any restore drill.
For #0010 this shrinks the work: the Storage Box + Borg pattern already exists and
is proven, so Gitea needs only its own repo there. The disabled cron (no backups
since 2026-07-30) remains separately urgent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified via DoH: _dmarc.axion1337.de is p=reject (sorb changed it), subdomains
inherit reject with sp= absent per RFC 7489, and the noted DKIM gap was a false
alarm — IONOS uses s1-ionos/s2-ionos/s42582890 selectors, all present with valid
keys. Apex SPF left at ~all deliberately: real mail flows over the apex and DMARC
already enforces reject, so -all adds little while risking silent send breakage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ftp.axion1337.de deleted and verified (NXDOMAIN). All four DNS hygiene issues
from today's batch (#0001, #0003, #0005, plus #0007 earlier) are now closed:
rohana and selendis hardened with Null-MX/SPF -all/DMARC reject, matrix and
www.game removed, ftp removed. Production A/AAAA records untouched throughout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
matrix.axion1337.de was already removed by sorb independently (platform runs
under .chat), so www.matrix went with it (confirmed NXDOMAIN via two
independent DoH resolvers). www.game deleted through IONOS and verified.
#0005 down to a single remaining item: ftp.axion1337.de.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both done step-by-step with sorb through the IONOS panel, externally confirmed
via DoH: rohana (www removed, Null-MX, SPF -all, DMARC reject — was previously
unprotected) and selendis (IONOS Mail service deactivated to unlock MX
deletion, 3 DKIM CNAMEs + www removed, SPF edited in place, Null-MX, DMARC
reject). Service-record lesson noted for the remaining matrix cleanup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Previous commit set three issues in-progress, tripping the WIP<=2 rule. The
verification is done; the DNS mutations are sorb's to run in IONOS, so #0001
and #0003 move to waiting (with wartegrund) while #0005 drives the batch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified current axion1337.de zone state via DoH: www.matrix/www.game are
IONOS-default records nothing serves (cluster routes .chat, no matching cert);
rohana is unhardened (no Null-MX/-all/reject) with www.rohana still present;
selendis mail-set untouched; matrix mail-set + autodiscover present; ftp is
IONOS-hosting ballast. Attached a consolidated per-name IONOS action list; the
mutations are sorb's to run in IONOS (no API access from here). Batch in-progress.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cert renewal switched to DNS-01 (IONOS): test issuance validated end-to-end
(LE YR1, valid to 2026-11-12), the shared letsencrypt resolver now renews
rohana/selendis via DNS-01, so the September renewal needs no open port 443.
Syncs the canonical file with the already-closed git.lab tracker issue 7.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Traefik stack is thread-net-git (manual compose deploy on CFGMON). DNS-01 diff
is ready (tlschallenge -> dnschallenge/ionos + IONOS_API_KEY via host .env).
Two human dependencies remain: create the IONOS API key and deploy+verify on
CFGMON (no SSH from here).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Consolidate the 13 findings + learnings from the Wiki.js AAR into
docs/wiki/stolpersteine/wikijs.md (config/deploy, theming, navigation,
locale/timezone incl. the standing fork patch, access control, git-storage),
link it from the wiki index, and set the AAR status to harvested.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Capture the follow-up work in the Wiki.js AAR: locale migration mechanics
(migrateToLocale only patches pages; rebuild tree+index; nav must move to the
new locale), the new-user timezone source (DB column default), API-only wiki
editing, and the standing upstream deviation (startup sed on users.js) with its
upgrade-check anchor. Stumbles 11-13 + a dated Nachtrag section.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
#0023 (Docusaurus navbar logo) is moot since Wiki.js replaced Docusaurus
(ADR-0014) -> rejected. #0007 (cert renewal, due 2026-09-28) gets a concrete
plan: pursue DNS-01 (approach B, already recommended) before mid-September,
with the port-opening fallback A as a dated calendar checkpoint.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sidebar links render href=target verbatim, so page targets need a leading
slash; without it they resolve relatively and 404 from any sub-path. Captured
as stumble #10 for forkers; fixed in gitops set_navigation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wiki.js→Gitea (sorb/ThreadNetWiki) is wired and verified end-to-end (status
operational, page create/delete propagates). Move ADR-0015 to accepted with an
implementation note; mark the git-storage part of #0048 done. Remaining: the
canonize job Gitea→git.lab (needs the target repo decision).
The cluster cannot reach git.lab (deliberate lab-independence), so #0048's
git.lab-repo-as-storage is infeasible. ADR-0015 routes Wiki.js content
cluster→Gitea→canonize to git.lab, reusing the accepted TURN-rotation pattern;
status proposed, pending sorb's ratification and two prerequisites (Gitea repo +
deploy PAT). Note the flagged contradiction on #0048 and the live theming
progress (logo, shared login background, dark default) on #0050. STATUS regen.
wiki.axion1337.chat (public, Let's-Encrypt TLS, native Authentik-OIDC login, no
forward-auth) added to #0048 (ingress/cert pattern) and #0049 (redirect URI, same
URL for user and admin, role decides). Kept out of ADR-0014 deliberately: accepted
ADRs are not edited, and the hostname is a deployment detail, not a new decision.
ADR-0014 records the decision: Wiki.js replaces Docusaurus for the platform wiki
— the only option meeting both hard requirements (per-group abschotten AND
docs-as-code in git). BookStack ruled out (DB-only, no git). Scope excludes
homelab/docs; neckbeard docs stay in management; content in a dedicated wiki repo
(not a branch, not a monorepo). ADR-0007 set to superseded. #0047 resolved
(decided: Wiki.js). Build issues 0048 (deploy + git storage), 0049 (OIDC + roles/
abschottung: admins write, users read-only), 0050 (theming, colours+logo extracted
from homelab/wiki). #0046 becomes the umbrella. STATUS regenerated; all gates green.
#0024 decided: axionwiki.lab (development-time). #0020 decided: stay on
Docusaurus with Authentik forward-auth, not the last word on the surface. Both
closed with the decision recorded. New follow-ups the user asked to keep:
0046 (move the wiki into the ThreadNet Server Suite, M4) and 0047 (re-examine
surface alternatives beyond BookStack afterwards, M2). STATUS regenerated;
validate, gen_status --check, upstream_drift and pruefe_prosa green.
The job got past the git fix but then failed the urllib call to https://git.lab
with CERTIFICATE_VERIFY_FAILED: gruppenpruefung.py uses urllib's default trust,
which in python:3.12-alpine does not include the private aXionLabs CA. Point
SSL_CERT_FILE at the repo's ci/lab-ca-chain.crt (the same chain curl --cacert
uses); Python honours it in the default SSL context. Verified locally: the
context loads the 2 lab CA certs.
gruppenpruefung.py runs 'git log --all' over the sibling clones (line 108) but
its job used python:3.12-alpine with no before_script, so it would hit the same
FileNotFoundError: 'git' as validate did. Dormant only because the job runs on
schedule/web, not push. Add 'apk add git'. The job's API/CA and GITLAB_TOKEN
prerequisites remain tracked separately (management#31).
pruefe_prosa.py shells out to 'git cat-file' to verify cited commit SHAs, but
the python:3.12-alpine image has no git and before_script only installed pyyaml.
The job crashed with FileNotFoundError on every push since the migration
(pipelines #255, #257). Add 'apk add git' to before_script. Verified green in
the same image locally: validate, gen_status --check, upstream_drift and
pruefe_prosa all pass.
Externalises what this session held that the migrated repo did not:
- vision/threadnet.md: the three capabilities that justify the forks beyond
rebranding (AV scanning into encrypted rooms, call-quality defaults with a
client-side-only privacy line, expiring guest access via @concierge).
- issue 0043 (M5): case-insensitive uniqueness in the matrix-invitation prompt
stage — the open residual of ADR-0011.
- issue 0044 (M5): auto-restart consumers on SOPS values-secret change — the
footgun behind the on_conflict fix sitting inactive until a manual restart.
- issue 0045 (M1): report_event.admin_message_md unset — content reports
dead-end with no contact path (verified still open against live config).
- sources/protokolle: the raw apo-call diagnosis history, including the four
ruled-out hypotheses and the harmful DB write, as the source behind the AAR.
STATUS.md regenerated (M5 appears for the first time). validate, gen_status
--check, upstream_drift and pruefe_prosa all green in the CI image.
Frozen handoff record for the upstream repo, English per neckbeard's
own artifact convention: the completed first size-L run (the ADR-0006
v1.0.0 trigger), eleven feedback items each with field evidence and
reference implementations, and a where-to-look table. Issue 0040 now
points at it; go-live item 1 in issue 0042 is ticked off by this push.
Size S under the granted exception - one deliverable, no design
decisions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The design doc closes with its AAR (planned/actual/why/learnings, the
six acceptance criteria checked off 6/6, the session's shakiest calls
named) and moves to docs/design/done/ with status done. Harvest: a
stolpersteine wiki page distilled from the AAR (hex is not a git SHA,
TZ on the git process, python floor, anonymous Gitea negatives,
negative tests, directory links), and the neckbeard feedback list
becomes issue 0040 - a deliberate separate act, per the design's
non-goals. Operational follow-up is issues 0041 (refine imported
wartegrund) and 0042 (go-live: push, first mirror run, CI schedule,
milestone for gitops#61). Final chain green: validate 0/0 over 36 open
issues, gen_status --check current, drift 0, prosa 0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gate 4, slice 5: eight component declarations under docs/components/
(filename = canonical slug, F-008 answered by construction; staged
dormancy of thread-net-git/threadnet-operating finally representable,
game-operating/gameserver as external with their field-test caveats),
five pointer-rollout follow-up issues (0035-0039, ADR-0013),
gruppenpruefung.py joins the stillstandspruefung family (runtime group
list vs declarations, pointer presence, group-wide milestone/priority
duty, issue drift, git hygiene since the 2026-08-07 rule boundary,
bot exception per ADR-0009) with its own scheduled CI job, and
spiegel_issues.py mirrors repo to GitLab (title, state, milestone,
priority, due, status label only - never descriptions, never
backwards, dry-run by default, GitLab-only issues are reported and
never auto-closed).
Verified - all four pattern demos fire (acceptance criterion 5, 4/4):
A) old CLAUDE.md claims M1-M4 while the frozen export knows M5;
B) hygiene over full clone history finds 222 real-clock commits by
own identities (matches the frozen Session-1 numbers per repo);
C) covered in slices 3/4 (five task blocks, now 0);
D) covered in slice 3 (orphaned SHA citations, now resolved/curated).
Live run (read-only): exactly the four missing pointers (F-011) and
gitops#61 without milestone as red findings, zero drift on all 26
mirrored issues, group list consistent. Mirror dry-run plans 7
creations, 0 updates, wrote nothing. Offline chain green: validate
0/0, gen_status --check current, drift 0, prosa 0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gate 4, slice 4: 26 open management issues imported from live git.lab
(read-only, descriptions included as authorized; GitLab iid = file id,
labels/milestone/priority/due/host/area mapped into frontmatter, the
import aborts instead of inventing a missing milestone or priority).
Two new issues close the F-004 gap where work was really still open
(0033 OVERMIND-01, 0034 CFGMON-11 incl. the plaintext npm-token
rotation); CFGMON-12/13 already route to verified git.lab issues,
MATRIX-05 is done and needs none (agreed with sorb). The three wiki
task blocks now reference their issues, roadmap.md hands all counts to
the generated STATUS.md and states M1-M5 per ADR-0010 (closing F-001
in the canonical prose), pruefe_prosa joins the CI validate job, and
the import protocol under docs/sources/migration/ records every
intervention into imported text.
Verified: validate 0/0 over 29 issue files, gen_status --check current
(distribution line M1 9 - M2 17 - M4 2 plus per-issue milestone and
priority), pruefe_prosa 0 errors with clones and 0 errors/15 unchecked
citations in offline CI mode, drift check green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gate 4, slice 3: verfahren/, hosts/, vision/ and shared/ moved via git
mv - six AARs to docs/aar/ (four harvested by the 2026-08-09 retro,
two open), procedures and host knowledge to docs/wiki/ (admin,
deployment, architecture, new area vision), the retro protocol and the
commit mapping table to docs/sources/ (protokolle/, migration/). New:
the wiki index linking every page, and the mirror-topology page
carrying the why-two-places reasoning verbatim from the old CLAUDE.md
(F-013 preserved). All moved-path references retargeted; the link
checker drove the sweep to zero.
pruefe_prosa.py added (pattern C+D): SHA citations resolve via repo,
mapping table, optional component clones or a curated exemption list
(documented dead Gitea-force-push commits, a vendor-repo tag, an
Authentik uid that is hex but no git SHA, the external neckbeard
reference); wiki task prose without an issue reference errors, with a
visible pragma for deliberate checklists; the dead-tracker denylist
now covers every mirrored repo's retired Gitea tracker (F-005) - two
links re-verified against live GitLab titles and retargeted, five
defused into honest historical citations.
Verified: validate 0/0, gen_status --check current, drift 0. Demo on
the pre-migration state fires 6 findings (3 orphaned SHAs, 3 task
blocks); on the current tree exactly the 3 F-004 task blocks remain -
they turn green in slice 4 when the issues exist, which is why
pruefe_prosa joins CI only then.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gate 4, slice 2: decisions/0001-0011 moved via git mv with schema
frontmatter prepended (status and date taken from each body's own
Status line - 0007 stays proposed, its decision is open in #20; bodies
unchanged except relative links gaining one directory level). The old
scheme's README and template retire - their rules already live in
AGENTS.md section 6 and the neckbeard ADR template. Every reference to
decisions/ across the tree retargeted (root files, not-yet-moved
verfahren/hosts/shared files, design doc and session ADR frontmatter).
Verified: validate 0 errors (11 ported + 2 session ADRs + duplicate-id
guard), gen_status --check current with all 13 ADRs listed, drift
check 0 findings, negative test shows a cloned id 0012 firing the
duplicate check.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tracer bullet of the migration design (Gate 4, slice 1): pinned v0.1.1
baseline under docs/sources/upstream/ with provenance note, the
Karpathy block moved verbatim to docs/sources/regelwerk/ (standing
rule mapped onto the sources read-only mechanism), AGENTS.md assembled
from the byte-true upstream sections plus the project section 6
(group rules condensed from the old CLAUDE.md), CLAUDE.md reduced to
the upstream pointer, WORKFLOW.md and all four templates copied,
schema.yaml extended (issue milestone/priority/status columns,
component type, wiki area vision - all flagged in the header),
validate.py and gen_status.py forked with marked extensions,
pruefe_upstream_drift.py added, STATUS.md generated, CI gains the
offline validate job, README directory link defused.
Verified: validate 0 errors 0 warnings (the three pre-existing
directory-link errors are gone), gen_status --check current,
drift check 0 findings, baseline byte-identical to the reference
checkout (10/10 files), four negative tests fire (WIP limit 3x
in-progress, waiting without wartegrund, component slug mismatch,
single-byte drift in WORKFLOW.md). gen_status needs Python >= 3.10
locally (write_text newline) - noted for the design AAR.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reality moved after the Gate-3 approval (flagged by the human, verified
read-only): main gained decisions/0011 plus a new AAR, gitops gained two
commits, and the live backlog shows gitops#61 without a milestone - the
first real break of the 100% milestone discipline. Session ADRs
renumbered to avoid the id collision, counts updated (11 old ADRs, 6
AARs), gruppenpruefung gains the group-wide milestone/priority duty
check backed by that real case. Addendum in the design doc records all
of it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Complete target file map, exact schema extensions, script signatures
without bodies, CI flow, per-check assertions including the four
pattern demonstrations and negative tests, DO NOT CHANGE boundaries,
and the six shakiest calls named.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Amendments decided with the human at the Gate-2 STOP: docs/sources/
gets a by-source-type taxonomy (regelwerk/upstream/protokolle/
migration, proposed by sorb), the pinned v0.1.1 originals become a
byte-compare baseline against silent framework-file rewrites, AGENTS.md
carries the change-only-with-sorb rule forward, and the issue import
may read descriptions via the token (read-only). ADR-0011 and ADR-0012
flipped to accepted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two-way harvest as mandated by the Session-1 handoff: failure patterns
of both approaches tabled with the mechanism that closes each, all
seven neckbeard gaps dispositioned (plus two new ones found this
session), and the old approach's proven value folded into the target
architecture. Two directional decisions filed as proposed ADRs: issues
live in-repo with GitLab as a deterministically mirrored view (0011),
group rules canonical here with pointer components and a checkable
components artifact (0012). Migration map, check architecture split
offline/runtime, constraints, upstream feedback candidates.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Problem statement built on the four drift patterns from the Session-1
field test, six numeric acceptance criteria, non-goals (no history
rewrite, no push, no component rollout, no forge-state destruction),
announcement paragraph. Gates 2-5 deliberately not pre-filled, per
WORKFLOW.md. Frontmatter validates against neckbeard v0.1.1 schema
with 0 errors for this file.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Answers recorded from the Gate 0 questions, asked and confirmed by the
human on 2026-08-11: language de, size-S exception granted, purpose and
audience as stated in the frontmatter. Validated against neckbeard
v0.1.1 schema.yaml (823a08c) with 0 errors, 0 warnings; the framework
files themselves enter this repo only after the two-way harvest mandated
by the Session-1 handoff.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Records on_conflict: fail as standing policy (identity provisioning never links
a new upstream identity to an existing local account), the residual prompt-stage
uniqueness check, and the SOPS-secret-needs-restart rule. Decided by sorb.
The SOPS secret updated via Flux but MAS kept the old config in memory until a
rollout restart. Records committed != deployed != active for the security fix.
Root cause proven end to end: a pre-Authentik account that lost its profiles
row (deactivate clears it, reactivate does not recreate it) crashes the
displayname write path, so it never gets a display name and the Element Call
widget never initialises. Fixed with a cross-checked INSERT; open_id_tokens
went 0 -> 6 and the call joined. Records the ruled-out suspects, what led to
the solution, and the lessons - chief among them: compare old accounts against
freshly provisioned ones, and reproduce in a cleartext room before blaming
crypto. Also notes the account-takeover finding (gitops#61) surfaced along the
way.
