Job hung indefinitely on git clone - public-internet reachability to
rohana.axion1337.de from inside pods is currently down (related to
Issue #41's firewall/private-network findings), and the node-level
/etc/hosts workaround only helps containerd's image pulls, not processes
running inside pods (they resolve via CoreDNS + their own /etc/hosts, not
the node's). hostAliases routes this job's git/API traffic over the
private Hetzner network directly - verified working with a real pod.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Scheduled rotation (Issue #38). New secret generated, re-encrypted with
the scoped rotation age key, checksum/rotated-at annotations bumped so
Flux restarts coturn + synapse-main on merge. Please review and merge.
Closes issue #38's automation half (architecture fix + first rotation
already landed in earlier commits this session). Monthly CronJob
(rohana.axion1337.de/sorb/axion-secret-rotation:v1 - alpine + git/sops/
jq/age) that:
- generates a new secret
- re-encrypts coturn-secret.yaml and synapse-turn-secret.yaml using the
scoped rotation-only age key (added as an additional recipient in an
earlier commit) - never touches the repo's master sops-age key
- bumps the turn-secret-checksum (HelmRelease annotation) and rotated-at
(coturn Deployment annotation) so merging actually restarts both
consumers, reusing the existing checksum-annotation pattern already in
this repo rather than inventing a new mechanism
- opens a Pull Request rather than pushing straight to main - a human
reviews and merges, keeping a checkpoint before production picks up new
credentials while still automating the tedious coordination work
Needs a Gitea PAT (repo write scope) filled into
turn-secret-rotation-secret.yaml's gitea-token key before first use.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds a second age recipient (dedicated keypair, private key never touches
the repo) to just these two files, so the upcoming rotation CronJob can
decrypt/re-encrypt them without needing the repo's master sops-age key,
which can decrypt every secret in the repo. Verified: master key still
decrypts both files; the new scoped key can decrypt both files but
correctly cannot decrypt an unrelated secret (authentik-secret.yaml).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Server-side apply left the old RollingUpdate defaults (maxSurge/maxUnavailable)
on the live object from before strategy.type was ever set explicitly, which
the API rejects when combined with type: Recreate. Explicit null clears it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Discovered live: the default RollingUpdate strategy tries to schedule the
new pod before removing the old one, which can never succeed for a
hostNetwork Deployment on a single-node cluster (both pods would need the
same node ports 3478/5349 simultaneously). Surfaced by the rotated-at
annotation bump from the previous commit getting stuck in
FailedScheduling. Recreate kills the old pod first, freeing the ports.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Issue #38 discussion surfaced a real bug: the TURN shared secret was
duplicated - correctly SOPS-encrypted in coturn-secret.yaml, but also
hardcoded in plaintext in synapse-values.yaml (a plain, non-SOPS
ConfigMap), visible in git history. Also found turn_user_lifetime is
86400000ms (24h), not "short-lived" as previously assumed - raising the
stakes of the leak somewhat.
Extracted the turn config block into its own dedicated SOPS-encrypted
Secret (synapse-turn-secret.yaml), wired via a second HelmRelease
valuesFrom entry (same pattern already used for ess-mas-values-secret).
Rotated the value while doing this, so the leaked plaintext secret is no
longer live anywhere.
Added checksum/rotated-at annotations (matrix-stack HelmRelease's
existing element-config-checksum patch gets a sibling turn-secret-checksum;
coturn's Deployment pod template gets a rotated-at annotation) so future
rotations actually restart both consumers - Kubernetes doesn't restart
running pods when a referenced Secret's content changes on its own.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Live testing showed pg_dump got "Connection refused" immediately at job
start, but a manual pg_isready/psql from an identically-labeled pod a few
seconds later succeeded fine every time. The cluster's NetworkPolicy
controller needs a brief moment to program a brand-new pod's IP into the
target's allowed ingress rules; a job that starts dumping instantly can
race that window. backup.sh now waits for pg_isready before each pg_dump
(up to 15 tries / ~30s) instead of assuming connectivity is ready at
container start.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes issues #6 and #15 (treated jointly, per combined analysis in their
comments). Two CronJobs, one per namespace, each pushing to its own Borg
repo on the newly booked Storage Box (u641795.your-storagebox.de:23, BX11
1TB) with its own repo passphrase - a leaked passphrase for one doesn't
expose the other:
- matrix: synapse-backup dumps the synapse + matrixauthenticationservice
DBs (shared postgres, existing chart-generated POSTGRES_ADMIN_PASSWORD)
and the Synapse media_store PVC (read-only mount)
- authentik: authentik-backup dumps the authentik DB (existing
authentik-credentials pg-password)
Custom image (rohana.axion1337.de/sorb/axion-backup:v1, postgres:17-alpine
+ borgbackup + openssh-client - matches the live Postgres major version
exactly, unlike Alpine's stock postgresql16-client) pushed as a public
package, same pattern as the existing threadnet-web image (no
imagePullSecret needed).
SSH host key pinned via known_hosts ConfigMap (captured via ssh-keyscan
ahead of time) rather than trusting on first connect in an unattended job.
Retention: 7 daily / 4 weekly / 6 monthly via borg prune. Schedule: 03:00
and 03:15, offset to avoid resource contention.
NetworkPolicy: added each backup job's pod as an allowed source to its
namespace's existing postgres ingress rule (matrix's allow-ingress-postgres,
authentik's allow-ingress-authentik-postgresql from #37). Egress already
unrestricted in both namespaces, so no change needed for the outbound SSH
connection to the Storage Box.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes issue #37. The Bitnami postgresql subchart's auto-generated
NetworkPolicy restricted the port (5432) but not the source - any pod in
any namespace could reach it, since additive NetworkPolicy rules can't
restrict an existing permissive one, only the chart itself controls it
(hence not touched during the original #10 rollout).
Disabled via postgresql.primary.networkPolicy.enabled: false and replaced
with a scoped policy allowing only authentik-server and authentik-worker
(same namespace) on 5432, matching the pattern already used for the
matrix namespace's postgres access. No metrics exporter is enabled for
this instance, so no monitoring-namespace rule is needed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes issue #36. The OAuth2 Provider ("Matrix Authentication Service")
and its Application ("matrix" slug) linking Authentik to MAS were
originally clicked together by hand in the UI and existed nowhere as
code - unlike the flow fixes already captured in authentik-blueprints.yaml.
Losing the Authentik DB would have meant re-creating this from scratch,
including a fresh client_secret that MAS would no longer match.
The client_secret is read via !Env from AUTHENTIK_MAS_OIDC_CLIENT_SECRET,
sourced from a new key in the existing SOPS-encrypted authentik-credentials
Secret, rather than inlined into the blueprint ConfigMap (which isn't
itself encrypted). Value used is the actual live secret already in use,
read directly from the running Authentik DB - not a new/rotated one, so
this changes nothing about the current MAS<->Authentik pairing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Almost every tracked file in the repo had drifted to mode 777 on disk
(only files created fresh this session were unaffected), and a chunk of
that drift had already been committed as spurious +x bits on plain
YAML/Markdown files (authentik.yaml, kustomization.yaml, coturn.yaml,
element-server-suite.yaml, TASKS.md, install.md, etc.) - none of these
need to be executable. Restored to 644 for regular files, 755 only for
actual scripts (postCreateCommand.sh, docker-init.sh, install-hooks.sh,
pre-commit hook, element-setup-linux.sh).
Also found element-setup-macos.command was missing +x despite having a
shebang and being meant for double-click execution on macOS - fixed.
Added .gitignore for .DS_Store and .claude/ and stopped tracking the five
.DS_Store files that had been committed by accident.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The devcontainer could never actually be built successfully - verified by
building it from scratch for the first time in a while. Found and fixed six
issues:
- kubectl: apt.kubernetes.io was deprecated/shut down by Google in 2023,
switched to the official successor repo pkgs.k8s.io
- docker-ce-cli: apt line hardcoded arch=amd64, breaking the build on
Apple Silicon; resolved dynamically via dpkg --print-architecture
- useradd -G docker failed because only the Docker CLI (no daemon) is
installed, so no package ever creates the docker group; added explicit
groupadd
- oh-my-zsh install had a nested-quoting bug that made the RUN step fail;
simplified to download-then-run instead of one nested `su -c "sh -c ..."`
- sops binary was hardcoded to linux.amd64, only working on arm64 by luck
via Docker Desktop's QEMU emulation; resolved dynamically like docker-ce
- docker.sock was mounted but unusable (permission denied) since the
container's docker group GID never matched the host socket's GID; added
a root entrypoint (docker-init.sh) that reconciles this at container
start, then drops to the vscode user via gosu
Also fixed two stale mas-secrets.sops.yaml references (actual filename is
mas-secret.yaml) in README.md and postCreateCommand.sh, set the vscode
user's default shell to zsh (oh-my-zsh was installed but never used by
default), and documented all of the above plus a build+run verification
snippet in README.md so this class of drift is caught before it goes
unnoticed again.
Verified end-to-end: cold `docker build --no-cache`, then a real container
run against the actual mounted kubeconfig, age key, and docker socket -
kubectl reaches the live cluster, sops decrypts a real secret, and docker
ps talks to the real daemon as the vscode user.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
install.md only covered generating a brand-new key during fresh setup.
Added the actual recovery path (retrieve the existing key from the
sops-age secret) - what today's session actually needed after the Mac
reinstall. Also flags the age key's single-backup-location weakness,
tracked in issue #20.
Created issues #11-#31 for the remaining backlog (VP9 retry, ThreadNet-Web
build bug, MAS template link, WAF, and 17 security/infra hardening items),
plus #13 for the previously-noted MAS custom-template idea. Removed/marked
done the stale checklist items that no longer reflected reality (Authentik
Stage 2/E2E-test/invite-links, Hetzner firewall, SSH hardening - all
already completed weeks ago). TASKS.md's detailed backlog section now
points to Gitea issues as the single source of truth instead of
duplicating tracking in two places.
H.264 uses classic simulcast (like VP8), not SVC, so it's compatible
with the fork's simulcast-shaped publish options without needing a
code fix - unlike VP9/AV1 (see the 2026-07-28 incident notes). Also
tends to be hardware-accelerated on more devices, notably iOS/Safari.
Server-side: LiveKit has no codec restriction configured, supports
H.264 by default.
User reported visible artifacts/blockiness during calls. Root cause:
without an explicit simulcast_layers config, the fork defaulted to
only [180p, 360p] fallback layers below the 1440p top layer - any
minor network hiccup caused a hard cliff straight to blocky 360p
instead of a graceful step-down. Added a 720p middle rung.
VP9 retry deferred: LiveKit uses SVC for vp9/av1 instead of classic
simulcast, and the fork's buildPublishOptions() always builds
simulcast-shaped layers regardless of codec - likely the real reason
forcing vp9 broke calls entirely. Needs a code fix before retrying.
Chart default of 20Mi request+limit OOM-killed the service today after
~74 days of uptime (3rd OOM over its lifetime), causing intermittent
call setup failures unrelated to the NetworkPolicy work happening in
parallel. 20Mi is razor-thin for a long-running Go service; bumping to
a still-modest 64Mi/128Mi request/limit.
Second port-config incident from this rollout: synapse calls MAS's
oauth2/introspect endpoint on port 8080 (not 8081 as assumed) for
every single authenticated request (sync, pushrules, capabilities,
etc). The rule only allowed kube-system (Traefik) on 8080, so every
authenticated API call failed with 503 "Unable to introspect the
access token" - all clients showed a lost connection. Patched live
immediately, this commit brings Git back in sync with that patch.
Ingress-only default-deny plus per-component allow rules, using named
container ports throughout (not Service ports) after the authentik
port-mismatch incident. Traefik -> element-web/element-admin/haproxy
(fronts both matrix.axion1337.chat and well-known)/MAS; synapse <-
haproxy+MAS; postgres <- synapse+MAS; RTC signalling via Traefik, RTC
media NodePorts left open to the internet by design; monitoring scrape
allows for synapse/postgres/rtc-sfu metrics; ACME solver allow. coturn
(hostNetwork) needs nothing - NetworkPolicy doesn't apply to it. Part
of issue #10.
Caused a live 502 immediately after deploy: the rule allowed 80/443
(the Service's external ports), but NetworkPolicy filters on the pod's
actual container port after kube-proxy's DNAT - authentik-server's
Service maps 80->9000 and 443->9443. Confirmed root cause by suspending
Flux reconciliation (it was silently re-applying my manual test
deletions) and testing with the policies truly absent.
Ingress-only default-deny (egress untouched) plus explicit allows:
Traefik (kube-system) + MAS (matrix ns) -> authentik-server on 80/443,
and Traefik -> cert-manager's ACME HTTP-01 solver pods on 8089.
authentik-postgresql already has its own Bitnami-chart-managed policy,
left alone. Part of issue #10.
v0.2.0 forced video_codec: vp9, which broke calls entirely (no audio/
video transmitted, despite server logs showing the codec regression
fallback to VP8 working). Root cause not fully confirmed. This build
keeps the 1440p/60fps/bitrate defaults but leaves video_codec unset
(defaults to vp8) to isolate whether the codec preference itself was
the trigger.
User reports calls transmit neither video nor audio with
v0.2.0-elementcall-mediaquality. Rolling back to known-good v0.1.0
while investigating; likely the video_codec: vp9 preference, not the
resolution/framerate bump.
Bumps elementWeb image to a build with the embedded Element Call
widget replaced by our own fork (rohana.axion1337.de/sorb/threadnet-call,
based on emmick4/element-call:livekit / upstream PR #3736), raising
default call quality to up to 1440p/60fps camera (VP9) and 1440p/30fps
screen share. These are seeded defaults users can still adjust in
Settings, not hard-enforced caps.
coturn started up cleanly every time (all TCP/UDP listeners on 3478/5349
opened fine per logs) but was killed every ~10s because the liveness
probe ran "netstat -uln", which doesn't exist in coturn/coturn:latest
(sh: netstat: not found) - 36000+ restarts over 88 days for a healthy
process. Switched to a tcpSocket probe against the TCP listener coturn
already opens on 3478, no in-container tooling required.
Neither Element Web's config.json nor MAS's branding config expose a
generic custom-link mechanism, and /if/user/ is blocked for type=external
Matrix accounts, so the working Authentik MFA setup flows had nowhere to
be discoverable. Added a docs/setup/security.html page, following the
exact existing pattern (ConfigMap -> initContainer cp -> nginx), linked
from index.html's Support section.
Several dead ends (TOTP/WebAuthn setup after an anonymous visit bounces
through login and lands on "/") trace back to Brand.default_application
being unset, which falls back to /if/user/ - blocked for type=external
Matrix accounts. Set it to the matrix Application. Only changes the bare
"/" fallback; explicit URLs like /if/admin/ are unaffected.
2FA is intentionally optional (not_configured_action=skip on the login
flow's validate stage). Users who opt in use the built-in single-stage
default-authenticator-totp-setup / -webauthn-setup flows directly, since
/if/user/ is blocked for type=external Matrix accounts. Both flows had
nothing after the setup stage, so completing enrollment fell back to
that same blocked dashboard. Appended the shared redirect stage.
matrix-recovery existed but had zero stage bindings, and the real login
flow (default-authentication-flow, MAS's authentication_flow) never
linked to it, so users had no working "forgot password" path. Reused
the same default-recovery-* stages the built-in default-recovery-flow
already uses successfully, added our redirect stage at the end, and
set default-authentication-identification.recovery_flow accordingly.
After the login stage, the flow had no destination, so it fell back to
authentik's own /if/user/ interface - which refuses type=external users
(the correct type for Matrix-only accounts), showing "Die Oberflaeche
kann nur von internen Nutzern geoeffnet werden". Added a static redirect
to https://axion1337.chat as the final stage.
The prompt stage had 16 unrelated system policies bound (OOBE, user
settings, recovery, etc.), likely from a "select all" slip while
configuring it manually. They crash when evaluated in an anonymous
enrollment context (AnonymousUser has no group_attributes, etc.),
surfacing as opaque errors after form submit. Cleared live and via
the blueprint so re-application doesn't reintroduce them.
The matrix-invitation flow only had Invite+Prompt stage bindings, both
at order=0 (undefined order), missing the Write/Password/Login stages
entirely — invited users were never written to the DB. Applied the fix
live in-cluster (mirroring the working matrix-enrollment stage chain),
and captured it as an Authentik Blueprint (ConfigMap, mounted via
blueprints.configMaps) so the flow state is reproducible via GitOps
instead of manual admin-UI clicks.
Move all Authentik troubleshooting guides into dedicated subdirectory:
- DIAGNOSTIK-AUTHENTIK-FLOW.md
- AUTHENTIK-FIX-TEMPLATE.md
- AUTHENTIK-INVITATION-FLOW-FIX.md
- AUTHENTIK-CREATE-INVITATION-FLOW.md
Add README.md with:
- Quick reference guide for each document
- Scenario-based navigation
- Known issues tracking
- Tips and best practices
This keeps the root directory clean and organizes related guides together.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Root cause analysis:
- Only one matrix-enrollment flow exists
- Used for both standard signup AND invitations
- Causes flow conflicts: 'Found existing plan for other flow'
- Error when Klaus enrollment attempted: 'kein ausstehender benutzer Anfrage wurde verweigert'
Solution:
- Create separate matrix-invitation flow
- Use for invitation links only
- Prevents conflicts and allows proper field capture (email is mandatory)
This guide provides:
1. Step-by-step flow creation (5 stages)
2. Field configuration for Prompt Stage
3. Binding setup for each stage
4. Testing procedure with invitation link
5. Troubleshooting checklist
Related issues: Klaus enrollment failure, Boje enrollment failure
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The Invitation Flow is not properly configured:
- Only username is captured during invitations
- Email field is missing (required for OIDC token generation)
- 'Fehler fehlende Rechte' error indicates incomplete user data
This guide walks through:
1. Diagnosing the current Invitation Flow configuration
2. Creating/repairing the Prompt Stage with email field
3. Adding the Prompt Stage to the Invitation Flow
4. Testing the complete enrollment process
Related to: User Boje enrollment failure via invitation link
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- DIAGNOSTIK-AUTHENTIK-FLOW.md: Comprehensive troubleshooting guide
- AUTHENTIK-FIX-TEMPLATE.md: Repair instructions for common issues
These guides help debug why Boje user was created in Authentik but not synchronized to Matrix.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Auto-Deploy on Push / verify-and-notify (push) Has been cancelled
- Add upstream_oauth2_config with Authentik provider credentials
- Configure OIDC with client_id and client_secret from Authentik
- Disable local password authentication (OIDC-only login)
- Set claims mapping: subject, localpart, displayname, email
This enables users to login via Authentik OIDC provider with email
and username claims properly mapped for Matrix user provisioning.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Auto-Deploy on Push / verify-and-notify (push) Has been cancelled
- deploy-on-push.yml: Verify YAML, check SOPS encryption, notify on deployments
- milestone-release.yml: Auto-create releases on milestone tags
Triggers:
- deploy-on-push: On any push to main (apps/clusters changes)
- milestone-release: On git tag m*-*-complete
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Thore CimbalandClaude Sonnet 5
turn-secret-rotation
Scrublord MacBadandClaude Haiku 4.5