The devcontainer could never actually be built successfully - verified by
building it from scratch for the first time in a while. Found and fixed six
issues:
- kubectl: apt.kubernetes.io was deprecated/shut down by Google in 2023,
switched to the official successor repo pkgs.k8s.io
- docker-ce-cli: apt line hardcoded arch=amd64, breaking the build on
Apple Silicon; resolved dynamically via dpkg --print-architecture
- useradd -G docker failed because only the Docker CLI (no daemon) is
installed, so no package ever creates the docker group; added explicit
groupadd
- oh-my-zsh install had a nested-quoting bug that made the RUN step fail;
simplified to download-then-run instead of one nested `su -c "sh -c ..."`
- sops binary was hardcoded to linux.amd64, only working on arm64 by luck
via Docker Desktop's QEMU emulation; resolved dynamically like docker-ce
- docker.sock was mounted but unusable (permission denied) since the
container's docker group GID never matched the host socket's GID; added
a root entrypoint (docker-init.sh) that reconciles this at container
start, then drops to the vscode user via gosu
Also fixed two stale mas-secrets.sops.yaml references (actual filename is
mas-secret.yaml) in README.md and postCreateCommand.sh, set the vscode
user's default shell to zsh (oh-my-zsh was installed but never used by
default), and documented all of the above plus a build+run verification
snippet in README.md so this class of drift is caught before it goes
unnoticed again.
Verified end-to-end: cold `docker build --no-cache`, then a real container
run against the actual mounted kubeconfig, age key, and docker socket -
kubectl reaches the live cluster, sops decrypts a real secret, and docker
ps talks to the real daemon as the vscode user.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
install.md only covered generating a brand-new key during fresh setup.
Added the actual recovery path (retrieve the existing key from the
sops-age secret) - what today's session actually needed after the Mac
reinstall. Also flags the age key's single-backup-location weakness,
tracked in issue #20.
Created issues #11-#31 for the remaining backlog (VP9 retry, ThreadNet-Web
build bug, MAS template link, WAF, and 17 security/infra hardening items),
plus #13 for the previously-noted MAS custom-template idea. Removed/marked
done the stale checklist items that no longer reflected reality (Authentik
Stage 2/E2E-test/invite-links, Hetzner firewall, SSH hardening - all
already completed weeks ago). TASKS.md's detailed backlog section now
points to Gitea issues as the single source of truth instead of
duplicating tracking in two places.
Auto-Deploy on Push / verify-and-notify (push) Canceled after 0s
H.264 uses classic simulcast (like VP8), not SVC, so it's compatible
with the fork's simulcast-shaped publish options without needing a
code fix - unlike VP9/AV1 (see the 2026-07-28 incident notes). Also
tends to be hardware-accelerated on more devices, notably iOS/Safari.
Server-side: LiveKit has no codec restriction configured, supports
H.264 by default.
Auto-Deploy on Push / verify-and-notify (push) Canceled after 0s
User reported visible artifacts/blockiness during calls. Root cause:
without an explicit simulcast_layers config, the fork defaulted to
only [180p, 360p] fallback layers below the 1440p top layer - any
minor network hiccup caused a hard cliff straight to blocky 360p
instead of a graceful step-down. Added a 720p middle rung.
VP9 retry deferred: LiveKit uses SVC for vp9/av1 instead of classic
simulcast, and the fork's buildPublishOptions() always builds
simulcast-shaped layers regardless of codec - likely the real reason
forcing vp9 broke calls entirely. Needs a code fix before retrying.
Auto-Deploy on Push / verify-and-notify (push) Canceled after 0s
Chart default of 20Mi request+limit OOM-killed the service today after
~74 days of uptime (3rd OOM over its lifetime), causing intermittent
call setup failures unrelated to the NetworkPolicy work happening in
parallel. 20Mi is razor-thin for a long-running Go service; bumping to
a still-modest 64Mi/128Mi request/limit.
Auto-Deploy on Push / verify-and-notify (push) Canceled after 0s
Second port-config incident from this rollout: synapse calls MAS's
oauth2/introspect endpoint on port 8080 (not 8081 as assumed) for
every single authenticated request (sync, pushrules, capabilities,
etc). The rule only allowed kube-system (Traefik) on 8080, so every
authenticated API call failed with 503 "Unable to introspect the
access token" - all clients showed a lost connection. Patched live
immediately, this commit brings Git back in sync with that patch.
Auto-Deploy on Push / verify-and-notify (push) Canceled after 0s
Ingress-only default-deny plus per-component allow rules, using named
container ports throughout (not Service ports) after the authentik
port-mismatch incident. Traefik -> element-web/element-admin/haproxy
(fronts both matrix.axion1337.chat and well-known)/MAS; synapse <-
haproxy+MAS; postgres <- synapse+MAS; RTC signalling via Traefik, RTC
media NodePorts left open to the internet by design; monitoring scrape
allows for synapse/postgres/rtc-sfu metrics; ACME solver allow. coturn
(hostNetwork) needs nothing - NetworkPolicy doesn't apply to it. Part
of issue #10.
Auto-Deploy on Push / verify-and-notify (push) Canceled after 0s
Caused a live 502 immediately after deploy: the rule allowed 80/443
(the Service's external ports), but NetworkPolicy filters on the pod's
actual container port after kube-proxy's DNAT - authentik-server's
Service maps 80->9000 and 443->9443. Confirmed root cause by suspending
Flux reconciliation (it was silently re-applying my manual test
deletions) and testing with the policies truly absent.
Auto-Deploy on Push / verify-and-notify (push) Canceled after 0s
Ingress-only default-deny (egress untouched) plus explicit allows:
Traefik (kube-system) + MAS (matrix ns) -> authentik-server on 80/443,
and Traefik -> cert-manager's ACME HTTP-01 solver pods on 8089.
authentik-postgresql already has its own Bitnami-chart-managed policy,
left alone. Part of issue #10.
Auto-Deploy on Push / verify-and-notify (push) Canceled after 0s
v0.2.0 forced video_codec: vp9, which broke calls entirely (no audio/
video transmitted, despite server logs showing the codec regression
fallback to VP8 working). Root cause not fully confirmed. This build
keeps the 1440p/60fps/bitrate defaults but leaves video_codec unset
(defaults to vp8) to isolate whether the codec preference itself was
the trigger.
Auto-Deploy on Push / verify-and-notify (push) Canceled after 0s
User reports calls transmit neither video nor audio with
v0.2.0-elementcall-mediaquality. Rolling back to known-good v0.1.0
while investigating; likely the video_codec: vp9 preference, not the
resolution/framerate bump.
Auto-Deploy on Push / verify-and-notify (push) Canceled after 0s
Bumps elementWeb image to a build with the embedded Element Call
widget replaced by our own fork (rohana.axion1337.de/sorb/threadnet-call,
based on emmick4/element-call:livekit / upstream PR #3736), raising
default call quality to up to 1440p/60fps camera (VP9) and 1440p/30fps
screen share. These are seeded defaults users can still adjust in
Settings, not hard-enforced caps.
Auto-Deploy on Push / verify-and-notify (push) Canceled after 0s
coturn started up cleanly every time (all TCP/UDP listeners on 3478/5349
opened fine per logs) but was killed every ~10s because the liveness
probe ran "netstat -uln", which doesn't exist in coturn/coturn:latest
(sh: netstat: not found) - 36000+ restarts over 88 days for a healthy
process. Switched to a tcpSocket probe against the TCP listener coturn
already opens on 3478, no in-container tooling required.
Auto-Deploy on Push / verify-and-notify (push) Canceled after 0s
Create Release on Milestone Tag / create-release (push) Canceled after 0s
Neither Element Web's config.json nor MAS's branding config expose a
generic custom-link mechanism, and /if/user/ is blocked for type=external
Matrix accounts, so the working Authentik MFA setup flows had nowhere to
be discoverable. Added a docs/setup/security.html page, following the
exact existing pattern (ConfigMap -> initContainer cp -> nginx), linked
from index.html's Support section.
Auto-Deploy on Push / verify-and-notify (push) Canceled after 0s
Several dead ends (TOTP/WebAuthn setup after an anonymous visit bounces
through login and lands on "/") trace back to Brand.default_application
being unset, which falls back to /if/user/ - blocked for type=external
Matrix accounts. Set it to the matrix Application. Only changes the bare
"/" fallback; explicit URLs like /if/admin/ are unaffected.
Auto-Deploy on Push / verify-and-notify (push) Canceled after 0s
2FA is intentionally optional (not_configured_action=skip on the login
flow's validate stage). Users who opt in use the built-in single-stage
default-authenticator-totp-setup / -webauthn-setup flows directly, since
/if/user/ is blocked for type=external Matrix accounts. Both flows had
nothing after the setup stage, so completing enrollment fell back to
that same blocked dashboard. Appended the shared redirect stage.
Auto-Deploy on Push / verify-and-notify (push) Canceled after 0s
matrix-recovery existed but had zero stage bindings, and the real login
flow (default-authentication-flow, MAS's authentication_flow) never
linked to it, so users had no working "forgot password" path. Reused
the same default-recovery-* stages the built-in default-recovery-flow
already uses successfully, added our redirect stage at the end, and
set default-authentication-identification.recovery_flow accordingly.
Auto-Deploy on Push / verify-and-notify (push) Canceled after 0s
After the login stage, the flow had no destination, so it fell back to
authentik's own /if/user/ interface - which refuses type=external users
(the correct type for Matrix-only accounts), showing "Die Oberflaeche
kann nur von internen Nutzern geoeffnet werden". Added a static redirect
to https://axion1337.chat as the final stage.
Auto-Deploy on Push / verify-and-notify (push) Canceled after 0s
The prompt stage had 16 unrelated system policies bound (OOBE, user
settings, recovery, etc.), likely from a "select all" slip while
configuring it manually. They crash when evaluated in an anonymous
enrollment context (AnonymousUser has no group_attributes, etc.),
surfacing as opaque errors after form submit. Cleared live and via
the blueprint so re-application doesn't reintroduce them.
Auto-Deploy on Push / verify-and-notify (push) Canceled after 0s
The matrix-invitation flow only had Invite+Prompt stage bindings, both
at order=0 (undefined order), missing the Write/Password/Login stages
entirely — invited users were never written to the DB. Applied the fix
live in-cluster (mirroring the working matrix-enrollment stage chain),
and captured it as an Authentik Blueprint (ConfigMap, mounted via
blueprints.configMaps) so the flow state is reproducible via GitOps
instead of manual admin-UI clicks.
Move all Authentik troubleshooting guides into dedicated subdirectory:
- DIAGNOSTIK-AUTHENTIK-FLOW.md
- AUTHENTIK-FIX-TEMPLATE.md
- AUTHENTIK-INVITATION-FLOW-FIX.md
- AUTHENTIK-CREATE-INVITATION-FLOW.md
Add README.md with:
- Quick reference guide for each document
- Scenario-based navigation
- Known issues tracking
- Tips and best practices
This keeps the root directory clean and organizes related guides together.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Root cause analysis:
- Only one matrix-enrollment flow exists
- Used for both standard signup AND invitations
- Causes flow conflicts: 'Found existing plan for other flow'
- Error when Klaus enrollment attempted: 'kein ausstehender benutzer Anfrage wurde verweigert'
Solution:
- Create separate matrix-invitation flow
- Use for invitation links only
- Prevents conflicts and allows proper field capture (email is mandatory)
This guide provides:
1. Step-by-step flow creation (5 stages)
2. Field configuration for Prompt Stage
3. Binding setup for each stage
4. Testing procedure with invitation link
5. Troubleshooting checklist
Related issues: Klaus enrollment failure, Boje enrollment failure
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The Invitation Flow is not properly configured:
- Only username is captured during invitations
- Email field is missing (required for OIDC token generation)
- 'Fehler fehlende Rechte' error indicates incomplete user data
This guide walks through:
1. Diagnosing the current Invitation Flow configuration
2. Creating/repairing the Prompt Stage with email field
3. Adding the Prompt Stage to the Invitation Flow
4. Testing the complete enrollment process
Related to: User Boje enrollment failure via invitation link
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- DIAGNOSTIK-AUTHENTIK-FLOW.md: Comprehensive troubleshooting guide
- AUTHENTIK-FIX-TEMPLATE.md: Repair instructions for common issues
These guides help debug why Boje user was created in Authentik but not synchronized to Matrix.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Auto-Deploy on Push / verify-and-notify (push) Has been cancelled
- Add upstream_oauth2_config with Authentik provider credentials
- Configure OIDC with client_id and client_secret from Authentik
- Disable local password authentication (OIDC-only login)
- Set claims mapping: subject, localpart, displayname, email
This enables users to login via Authentik OIDC provider with email
and username claims properly mapped for Matrix user provisioning.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Auto-Deploy on Push / verify-and-notify (push) Has been cancelled
- deploy-on-push.yml: Verify YAML, check SOPS encryption, notify on deployments
- milestone-release.yml: Auto-create releases on milestone tags
Triggers:
- deploy-on-push: On any push to main (apps/clusters changes)
- milestone-release: On git tag m*-*-complete
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Add at top of TASKS.md:
- Status Summary table (quick view of progress)
- Priority distribution (CRITICAL/HIGH/MEDIUM/LOW)
- Next Steps section (this week, 1-2 weeks)
- Each task includes: description, effort estimate, blocking factors
Makes task list immediately actionable and progress trackable.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Create docs/deployment-guides/ directory with comprehensive README covering:
- Deployment order for all 5 major features
- Architecture overview
- Critical values & configuration
- Links to detailed guides
Full documentation files:
- 01-turn-server-setup.md
- 02-authentik-identity-provider.md
- 03-monitoring-integration.md
- 04-element-customization.md
- 05-room-policies.md
To be added in follow-up commits.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Dockerfile with all required tools (kubectl, flux, helm, sops, age, etc.)
- devcontainer.json with VSCode config and extensions
- postCreateCommand.sh for setup verification
- Comprehensive README with setup instructions for macOS, Windows/WSL2, Linux
- Automatic mounts for kubeconfig, SSH keys, age encryption keys
- SOPS_AGE_KEY_FILE and KUBECONFIG pre-configured
Enables development on Windows, macOS, and Linux with consistent environment.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The config.json was truncated and had unclosed brackets. This prevented
Helm from properly merging the ElementWeb configuration, so the custom themes
were never loaded into the cluster.
This fix:
- Closes the unclosed JSON brackets
- Validates the full JSON structure
- Removes duplicate/extra closing brackets
- Ensures all 6 custom themes are properly included
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Add pre-commit hook (scripts/hooks/pre-commit) that automatically calculates
MD5 checksums of ConfigMaps (element-values.yaml, synapse-values.yaml)
- Update annotations in kustomization.yaml to trigger Flux CD HelmRelease syncs
- Add install script (scripts/install-hooks.sh) for easy hook setup
- Add comprehensive documentation (docs/ops-configmap-sync.md) explaining:
* Why Flux doesn't auto-detect ConfigMap changes
* How the checksum-based workaround works
* How to install and use the hook
* Troubleshooting and manual sync procedures
- Update README.md with post-clone hook installation step
This solves the issue where Flux CD doesn't automatically re-deploy when external
ConfigMaps are modified. Users no longer need manual checksum updates.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Reduce HelmRelease interval from 1h to 5m for faster sync
- Add checksum annotation to trigger reconciliation when element-values.yaml changes
- This ensures Flux CD re-deploys the chart when themes/config updates are made
To update the checksum after editing element-values.yaml:
md5sum apps/production/custom-configs/element-values.yaml
# Update the value in apps/production/kustomization.yaml patches[0].patch
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Thore CimbalandClaude Sonnet 5
Scrublord MacBadandClaude Haiku 4.5
sorb