Scheduled rotation (Issue #38). New secret generated, re-encrypted with
the scoped rotation age key, checksum/rotated-at annotations bumped so
Flux restarts coturn + synapse-main on merge. Please review and merge.
Das prometheus-node-exporter-HelmRelease (hostNetwork) kollidierte auf
dem Host mit dem etablierten systemd-Exporter auf Port 9100 und konnte
seit Anlage nie binden (4883 Restarts, CrashLoopBackOff). Host-Metriken
kommen weiterhin vom systemd-Dienst, den CFGMON direkt via 10.0.0.2:9100
scrapt - das DaemonSet haette sie nur dupliziert. Alloy-Scrape auf den
verwaisten Service ebenfalls entfernt.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PKhFj1S3UdD6xL2fbWPeYj
Der Service matrix-stack-synapse-main exponiert 8008/9093/8080/9001
(synapse-metrics = 9001) - Port 9000 existiert nicht, der Scrape war
seit Anlage tot. Aufgedeckt durch den ersten TargetDown-Alarm des neuen
Alertings (gitops#32) direkt nach dem Deploy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PKhFj1S3UdD6xL2fbWPeYj
Der deploy-on-push-Check (YAML-Manifest-Pruefung, SOPS-ENC-Kontrolle) laeuft
jetzt als leichter Job in der Lab-GitLab-CI; milestone-release.yml war seit
jeher toter Code (kein Runner, Releases wurden manuell per API erstellt,
siehe Issue #33). Flux bleibt unberuehrt - es zieht weiterhin vom
Gitea-Mirror. Damit verliert der Gitea-Runner builder-1 seinen letzten
aktiven Konsumenten.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Documentation audit across the repo found several places still describing
finished work as pending or in-progress (Authentik Stage 2, Element Call
fork, NetworkPolicies in TASKS.md's own Next Steps section, the Boje
troubleshooting entry). Also moves CLAUDE.md from the untracked parent
directory into the repo root and brings its content up to date, and
documents the new host-config/ pattern in README.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
unattended-upgrades was already active on the host, just never documented
or closed. Adds a generic, reusable systemd timer + script that fires
before the daily update window and notifies via email and a Matrix thread
reply if any packages are actually pending - reusing the mas-cli bot
account pattern established for Draupnir.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
First full rebuild deployed to production (previous releases used the
widgets/element-call/-only patch workaround, fixed but not cut over in
Issue #12). Includes the new client-side scan hooks (DecryptFile.ts,
ContentMessages.ts) that cover encrypted rooms, which the server-side
Synapse module (Issue #19) structurally cannot see.
Synapse's own media-scanning module (Issue #19) can never see E2EE
attachment content - a structural limit, not a bug. This adds a small
standalone HTTP wrapper around the same ClamAV instance, reachable
from browser JS at /_scan, so the ThreadNet-Web client fork can scan
plaintext both before encrypting/uploading and after downloading/
decrypting - covering both directions regardless of room encryption.
Auth via Synapse's own /whoami endpoint, no separate auth system.
Synapse runs on Twisted's reactor, not asyncio's event loop - the
original asyncio.open_connection/wait_for calls failed immediately
with "RuntimeError: no running event loop", silently fail-opening
every scan (confirmed live: EICAR test file passed through unscanned).
Rewritten using twisted.internet.endpoints.HostnameEndpoint/
connectProtocol and a custom Protocol for the INSTREAM conversation.
Deploys ClamAV and a small stdlib-only Synapse spam-checker module
implementing check_media_file_for_spam over clamd's INSTREAM protocol.
Unlike the originally-considered matrix-content-scanner proxy (which
needs client-side cooperation neither Element Web nor Element X
provide), this hooks Synapse's own module API directly - transparent
to every client for unencrypted media. No custom Synapse image needed:
the module is mounted via a ConfigMap onto PYTHONPATH using the ESS
chart's extraVolumes/extraVolumeMounts/extraEnv support. Fails open on
scanner errors so a ClamAV outage can't block all uploads.
Testing E2EE support for the management room per user request - marked
"not considered production safe" upstream, verifying no regressions
before deciding to keep it.
Confirmed by extracting dist/config.js from the actual image - v3.x's
getConfigPath() only checks --draupnir-config/--mjolnir-config CLI
flags, no longer NODE_CONFIG_DIR/node-config auto-discovery.
initialManager's automatic management-room creation (no manual room
setup needed) only works on v3.1.0+ - v2.9.0 crashed on first startup
trying to join a room that was never created.
Bot account registered via mas-cli, compatibility token issued and
stored via sops. Adds an explicit NetworkPolicy allow-rule since
Draupnir needs to reach Synapse's client-server API in-namespace,
which the existing default-deny policy would otherwise silently block.
Real testing (Safari + Firefox, fresh rejoins) confirmed VP9 selection
always falls back to VP8 - our fork's buildPublishOptions() forces
simulcast:true unconditionally, which breaks LiveKit's SVC negotiation
path that vp9/av1 require. This is the same failure mode originally
suspected in the 2026-07-28 incident; the "LiveKit handles this
automatically" assumption from doc research did not hold up in
practice. Reverting to the known-safe VP8/H.264/H.265 SFU allow-list
and widget image. Root cause documented in Issue #11 for a future,
scoped fix (conditional simulcast vs. scalabilityMode per codec).
Patches the re-enabled VP9/AV1 codec dropdown (threadnet-call) onto the
existing v0.1.0 base image. Test-only tag - to be reverted or made
permanent depending on real call test results.
Appends video/VP9 and video/AV1 to the SFU's room.enabled_codecs
allow-list via matrixRTC.sfu.additional, keeping all currently
negotiated codecs unchanged. Opt-in test per Issue #11 - codecs
become selectable, not the new default.
Both docs/TASKS.md and the deployment guide still described the
full-rebuild blockers as an open, unfixed bug - only the wiki mirror had
been updated when #12 was actually closed. Added the fix summary to
both, plus a note about today's Video-tab/i18n/codec-list Element Call
changes in the deployment guide (previously only tracked in the wiki).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Live-verified via matrix-rtc-sfu logs: VP9 gets silently rejected and
falls back to VP8 (SFU's enabledPublishCodecs is [VP8, H264, H265], no
VP9/AV1). Dropdown now only offers those three; also fixed
screenShareCodec's default which was "vp9" (fork commit ac99cc65 in
threadnet-call).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The MediaQualitySettings labels moved into the Video tab had no German
translations at all (en/app.json had all 10 keys, de/app.json had none) -
fork commit f13c4b43 in threadnet-call.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Moved from behind the Developer-mode toggle into the regular Video
settings tab (fork commit f61650c0 in threadnet-call) - these are just
resolution/framerate/bitrate/codec pickers, not actually developer-only
functionality, just accidentally buried where most users would never
find them.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
README described only the initial v0.1.0-era bootstrap - no mention of
Authentik, coturn/TURN, monitoring, NetworkPolicies, backups, or the
Element Web/Call forks, all of which have been live for months. Also
fixed: broken TOC links (pointed to Google search instead of anchors),
stale mas-secrets.sops.yaml / ess-mas-custom-secrets references (actual
names are mas-secret.yaml / ess-mas-values-secret).
Added pointers to CLAUDE.md, docs/TASKS.md, Gitea Releases (new SemVer
scheme), the wiki, and deployment guides, plus a note recommending the
devcontainer as the primary way to get local tooling.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
todo-tree stopped triggering on docs/TASKS.md because the file was
restructured (2026-07-28 backlog migration) to use plain markdown
checkboxes with no literal TODO/FIXME/etc. keywords, which is all
todo-tree's default config matches on. Added the documented
todo-tree.regex.regex + [ ]/[x] tags configuration (see
Gruntfuggly/todo-tree wiki) so it actually detects checkbox items, plus
red/green highlighting for open vs done.
Also cleaned up 19 stale open checkbox items left behind by that same
migration - they duplicated content already tracked as individual Gitea
issues (in old pre-migration detail, not the established "-> Issue #N"
pointer format the rest of the file already uses), including two
(Database Backup Strategy, Synapse Media PVC Backups) for issues that
are actually already closed. Converted all to pointer format or removed
where closed. Replaced the stale M1-M7 milestone table (contradicted its
own file header - said M4 "In Progress" while the summary line above
already said 0 in progress) with a pointer to the new SemVer Releases.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Live-tested Issue #38's rotation: after merging the automated rotation
PR, coturn restarted quickly (Kustomization-level, 1m interval), but
synapse-main lagged behind since it depends on this separate HelmRelease
with its own 5m interval - a real (self-healing, but avoidable) window
where coturn had the new TURN secret and Synapse still had the old one,
which would reject each other's credentials/relayed media. Matching the
interval to production-apps's 1m tightens that window without needing
any new automation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Job hung indefinitely on git clone - public-internet reachability to
rohana.axion1337.de from inside pods is currently down (related to
Issue #41's firewall/private-network findings), and the node-level
/etc/hosts workaround only helps containerd's image pulls, not processes
running inside pods (they resolve via CoreDNS + their own /etc/hosts, not
the node's). hostAliases routes this job's git/API traffic over the
private Hetzner network directly - verified working with a real pod.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Scheduled rotation (Issue #38). New secret generated, re-encrypted with
the scoped rotation age key, checksum/rotated-at annotations bumped so
Flux restarts coturn + synapse-main on merge. Please review and merge.
Closes issue #38's automation half (architecture fix + first rotation
already landed in earlier commits this session). Monthly CronJob
(rohana.axion1337.de/sorb/axion-secret-rotation:v1 - alpine + git/sops/
jq/age) that:
- generates a new secret
- re-encrypts coturn-secret.yaml and synapse-turn-secret.yaml using the
scoped rotation-only age key (added as an additional recipient in an
earlier commit) - never touches the repo's master sops-age key
- bumps the turn-secret-checksum (HelmRelease annotation) and rotated-at
(coturn Deployment annotation) so merging actually restarts both
consumers, reusing the existing checksum-annotation pattern already in
this repo rather than inventing a new mechanism
- opens a Pull Request rather than pushing straight to main - a human
reviews and merges, keeping a checkpoint before production picks up new
credentials while still automating the tedious coordination work
Needs a Gitea PAT (repo write scope) filled into
turn-secret-rotation-secret.yaml's gitea-token key before first use.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a second age recipient (dedicated keypair, private key never touches
the repo) to just these two files, so the upcoming rotation CronJob can
decrypt/re-encrypt them without needing the repo's master sops-age key,
which can decrypt every secret in the repo. Verified: master key still
decrypts both files; the new scoped key can decrypt both files but
correctly cannot decrypt an unrelated secret (authentik-secret.yaml).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Server-side apply left the old RollingUpdate defaults (maxSurge/maxUnavailable)
on the live object from before strategy.type was ever set explicitly, which
the API rejects when combined with type: Recreate. Explicit null clears it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Discovered live: the default RollingUpdate strategy tries to schedule the
new pod before removing the old one, which can never succeed for a
hostNetwork Deployment on a single-node cluster (both pods would need the
same node ports 3478/5349 simultaneously). Surfaced by the rotated-at
annotation bump from the previous commit getting stuck in
FailedScheduling. Recreate kills the old pod first, freeing the ports.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Issue #38 discussion surfaced a real bug: the TURN shared secret was
duplicated - correctly SOPS-encrypted in coturn-secret.yaml, but also
hardcoded in plaintext in synapse-values.yaml (a plain, non-SOPS
ConfigMap), visible in git history. Also found turn_user_lifetime is
86400000ms (24h), not "short-lived" as previously assumed - raising the
stakes of the leak somewhat.
Extracted the turn config block into its own dedicated SOPS-encrypted
Secret (synapse-turn-secret.yaml), wired via a second HelmRelease
valuesFrom entry (same pattern already used for ess-mas-values-secret).
Rotated the value while doing this, so the leaked plaintext secret is no
longer live anywhere.
Added checksum/rotated-at annotations (matrix-stack HelmRelease's
existing element-config-checksum patch gets a sibling turn-secret-checksum;
coturn's Deployment pod template gets a rotated-at annotation) so future
rotations actually restart both consumers - Kubernetes doesn't restart
running pods when a referenced Secret's content changes on its own.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Live testing showed pg_dump got "Connection refused" immediately at job
start, but a manual pg_isready/psql from an identically-labeled pod a few
seconds later succeeded fine every time. The cluster's NetworkPolicy
controller needs a brief moment to program a brand-new pod's IP into the
target's allowed ingress rules; a job that starts dumping instantly can
race that window. backup.sh now waits for pg_isready before each pg_dump
(up to 15 tries / ~30s) instead of assuming connectivity is ready at
container start.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes issues #6 and #15 (treated jointly, per combined analysis in their
comments). Two CronJobs, one per namespace, each pushing to its own Borg
repo on the newly booked Storage Box (u641795.your-storagebox.de:23, BX11
1TB) with its own repo passphrase - a leaked passphrase for one doesn't
expose the other:
- matrix: synapse-backup dumps the synapse + matrixauthenticationservice
DBs (shared postgres, existing chart-generated POSTGRES_ADMIN_PASSWORD)
and the Synapse media_store PVC (read-only mount)
- authentik: authentik-backup dumps the authentik DB (existing
authentik-credentials pg-password)
Custom image (rohana.axion1337.de/sorb/axion-backup:v1, postgres:17-alpine
+ borgbackup + openssh-client - matches the live Postgres major version
exactly, unlike Alpine's stock postgresql16-client) pushed as a public
package, same pattern as the existing threadnet-web image (no
imagePullSecret needed).
SSH host key pinned via known_hosts ConfigMap (captured via ssh-keyscan
ahead of time) rather than trusting on first connect in an unattended job.
Retention: 7 daily / 4 weekly / 6 monthly via borg prune. Schedule: 03:00
and 03:15, offset to avoid resource contention.
NetworkPolicy: added each backup job's pod as an allowed source to its
namespace's existing postgres ingress rule (matrix's allow-ingress-postgres,
authentik's allow-ingress-authentik-postgresql from #37). Egress already
unrestricted in both namespaces, so no change needed for the outbound SSH
connection to the Storage Box.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes issue #37. The Bitnami postgresql subchart's auto-generated
NetworkPolicy restricted the port (5432) but not the source - any pod in
any namespace could reach it, since additive NetworkPolicy rules can't
restrict an existing permissive one, only the chart itself controls it
(hence not touched during the original #10 rollout).
Disabled via postgresql.primary.networkPolicy.enabled: false and replaced
with a scoped policy allowing only authentik-server and authentik-worker
(same namespace) on 5432, matching the pattern already used for the
matrix namespace's postgres access. No metrics exporter is enabled for
this instance, so no monitoring-namespace rule is needed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Thore Cimbal
turn-secret-rotation
