Live testing showed pg_dump got "Connection refused" immediately at job
start, but a manual pg_isready/psql from an identically-labeled pod a few
seconds later succeeded fine every time. The cluster's NetworkPolicy
controller needs a brief moment to program a brand-new pod's IP into the
target's allowed ingress rules; a job that starts dumping instantly can
race that window. backup.sh now waits for pg_isready before each pg_dump
(up to 15 tries / ~30s) instead of assuming connectivity is ready at
container start.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes issues #6 and #15 (treated jointly, per combined analysis in their
comments). Two CronJobs, one per namespace, each pushing to its own Borg
repo on the newly booked Storage Box (u641795.your-storagebox.de:23, BX11
1TB) with its own repo passphrase - a leaked passphrase for one doesn't
expose the other:
- matrix: synapse-backup dumps the synapse + matrixauthenticationservice
DBs (shared postgres, existing chart-generated POSTGRES_ADMIN_PASSWORD)
and the Synapse media_store PVC (read-only mount)
- authentik: authentik-backup dumps the authentik DB (existing
authentik-credentials pg-password)
Custom image (rohana.axion1337.de/sorb/axion-backup:v1, postgres:17-alpine
+ borgbackup + openssh-client - matches the live Postgres major version
exactly, unlike Alpine's stock postgresql16-client) pushed as a public
package, same pattern as the existing threadnet-web image (no
imagePullSecret needed).
SSH host key pinned via known_hosts ConfigMap (captured via ssh-keyscan
ahead of time) rather than trusting on first connect in an unattended job.
Retention: 7 daily / 4 weekly / 6 monthly via borg prune. Schedule: 03:00
and 03:15, offset to avoid resource contention.
NetworkPolicy: added each backup job's pod as an allowed source to its
namespace's existing postgres ingress rule (matrix's allow-ingress-postgres,
authentik's allow-ingress-authentik-postgresql from #37). Egress already
unrestricted in both namespaces, so no change needed for the outbound SSH
connection to the Storage Box.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes issue #37. The Bitnami postgresql subchart's auto-generated
NetworkPolicy restricted the port (5432) but not the source - any pod in
any namespace could reach it, since additive NetworkPolicy rules can't
restrict an existing permissive one, only the chart itself controls it
(hence not touched during the original #10 rollout).
Disabled via postgresql.primary.networkPolicy.enabled: false and replaced
with a scoped policy allowing only authentik-server and authentik-worker
(same namespace) on 5432, matching the pattern already used for the
matrix namespace's postgres access. No metrics exporter is enabled for
this instance, so no monitoring-namespace rule is needed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes issue #36. The OAuth2 Provider ("Matrix Authentication Service")
and its Application ("matrix" slug) linking Authentik to MAS were
originally clicked together by hand in the UI and existed nowhere as
code - unlike the flow fixes already captured in authentik-blueprints.yaml.
Losing the Authentik DB would have meant re-creating this from scratch,
including a fresh client_secret that MAS would no longer match.
The client_secret is read via !Env from AUTHENTIK_MAS_OIDC_CLIENT_SECRET,
sourced from a new key in the existing SOPS-encrypted authentik-credentials
Secret, rather than inlined into the blueprint ConfigMap (which isn't
itself encrypted). Value used is the actual live secret already in use,
read directly from the running Authentik DB - not a new/rotated one, so
this changes nothing about the current MAS<->Authentik pairing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Almost every tracked file in the repo had drifted to mode 777 on disk
(only files created fresh this session were unaffected), and a chunk of
that drift had already been committed as spurious +x bits on plain
YAML/Markdown files (authentik.yaml, kustomization.yaml, coturn.yaml,
element-server-suite.yaml, TASKS.md, install.md, etc.) - none of these
need to be executable. Restored to 644 for regular files, 755 only for
actual scripts (postCreateCommand.sh, docker-init.sh, install-hooks.sh,
pre-commit hook, element-setup-linux.sh).
Also found element-setup-macos.command was missing +x despite having a
shebang and being meant for double-click execution on macOS - fixed.
Added .gitignore for .DS_Store and .claude/ and stopped tracking the five
.DS_Store files that had been committed by accident.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
H.264 uses classic simulcast (like VP8), not SVC, so it's compatible
with the fork's simulcast-shaped publish options without needing a
code fix - unlike VP9/AV1 (see the 2026-07-28 incident notes). Also
tends to be hardware-accelerated on more devices, notably iOS/Safari.
Server-side: LiveKit has no codec restriction configured, supports
H.264 by default.
User reported visible artifacts/blockiness during calls. Root cause:
without an explicit simulcast_layers config, the fork defaulted to
only [180p, 360p] fallback layers below the 1440p top layer - any
minor network hiccup caused a hard cliff straight to blocky 360p
instead of a graceful step-down. Added a 720p middle rung.
VP9 retry deferred: LiveKit uses SVC for vp9/av1 instead of classic
simulcast, and the fork's buildPublishOptions() always builds
simulcast-shaped layers regardless of codec - likely the real reason
forcing vp9 broke calls entirely. Needs a code fix before retrying.
Chart default of 20Mi request+limit OOM-killed the service today after
~74 days of uptime (3rd OOM over its lifetime), causing intermittent
call setup failures unrelated to the NetworkPolicy work happening in
parallel. 20Mi is razor-thin for a long-running Go service; bumping to
a still-modest 64Mi/128Mi request/limit.
Second port-config incident from this rollout: synapse calls MAS's
oauth2/introspect endpoint on port 8080 (not 8081 as assumed) for
every single authenticated request (sync, pushrules, capabilities,
etc). The rule only allowed kube-system (Traefik) on 8080, so every
authenticated API call failed with 503 "Unable to introspect the
access token" - all clients showed a lost connection. Patched live
immediately, this commit brings Git back in sync with that patch.
Ingress-only default-deny plus per-component allow rules, using named
container ports throughout (not Service ports) after the authentik
port-mismatch incident. Traefik -> element-web/element-admin/haproxy
(fronts both matrix.axion1337.chat and well-known)/MAS; synapse <-
haproxy+MAS; postgres <- synapse+MAS; RTC signalling via Traefik, RTC
media NodePorts left open to the internet by design; monitoring scrape
allows for synapse/postgres/rtc-sfu metrics; ACME solver allow. coturn
(hostNetwork) needs nothing - NetworkPolicy doesn't apply to it. Part
of issue #10.
Caused a live 502 immediately after deploy: the rule allowed 80/443
(the Service's external ports), but NetworkPolicy filters on the pod's
actual container port after kube-proxy's DNAT - authentik-server's
Service maps 80->9000 and 443->9443. Confirmed root cause by suspending
Flux reconciliation (it was silently re-applying my manual test
deletions) and testing with the policies truly absent.
Ingress-only default-deny (egress untouched) plus explicit allows:
Traefik (kube-system) + MAS (matrix ns) -> authentik-server on 80/443,
and Traefik -> cert-manager's ACME HTTP-01 solver pods on 8089.
authentik-postgresql already has its own Bitnami-chart-managed policy,
left alone. Part of issue #10.
v0.2.0 forced video_codec: vp9, which broke calls entirely (no audio/
video transmitted, despite server logs showing the codec regression
fallback to VP8 working). Root cause not fully confirmed. This build
keeps the 1440p/60fps/bitrate defaults but leaves video_codec unset
(defaults to vp8) to isolate whether the codec preference itself was
the trigger.
User reports calls transmit neither video nor audio with
v0.2.0-elementcall-mediaquality. Rolling back to known-good v0.1.0
while investigating; likely the video_codec: vp9 preference, not the
resolution/framerate bump.
Bumps elementWeb image to a build with the embedded Element Call
widget replaced by our own fork (rohana.axion1337.de/sorb/threadnet-call,
based on emmick4/element-call:livekit / upstream PR #3736), raising
default call quality to up to 1440p/60fps camera (VP9) and 1440p/30fps
screen share. These are seeded defaults users can still adjust in
Settings, not hard-enforced caps.
coturn started up cleanly every time (all TCP/UDP listeners on 3478/5349
opened fine per logs) but was killed every ~10s because the liveness
probe ran "netstat -uln", which doesn't exist in coturn/coturn:latest
(sh: netstat: not found) - 36000+ restarts over 88 days for a healthy
process. Switched to a tcpSocket probe against the TCP listener coturn
already opens on 3478, no in-container tooling required.
Neither Element Web's config.json nor MAS's branding config expose a
generic custom-link mechanism, and /if/user/ is blocked for type=external
Matrix accounts, so the working Authentik MFA setup flows had nowhere to
be discoverable. Added a docs/setup/security.html page, following the
exact existing pattern (ConfigMap -> initContainer cp -> nginx), linked
from index.html's Support section.
Several dead ends (TOTP/WebAuthn setup after an anonymous visit bounces
through login and lands on "/") trace back to Brand.default_application
being unset, which falls back to /if/user/ - blocked for type=external
Matrix accounts. Set it to the matrix Application. Only changes the bare
"/" fallback; explicit URLs like /if/admin/ are unaffected.
2FA is intentionally optional (not_configured_action=skip on the login
flow's validate stage). Users who opt in use the built-in single-stage
default-authenticator-totp-setup / -webauthn-setup flows directly, since
/if/user/ is blocked for type=external Matrix accounts. Both flows had
nothing after the setup stage, so completing enrollment fell back to
that same blocked dashboard. Appended the shared redirect stage.
matrix-recovery existed but had zero stage bindings, and the real login
flow (default-authentication-flow, MAS's authentication_flow) never
linked to it, so users had no working "forgot password" path. Reused
the same default-recovery-* stages the built-in default-recovery-flow
already uses successfully, added our redirect stage at the end, and
set default-authentication-identification.recovery_flow accordingly.
After the login stage, the flow had no destination, so it fell back to
authentik's own /if/user/ interface - which refuses type=external users
(the correct type for Matrix-only accounts), showing "Die Oberflaeche
kann nur von internen Nutzern geoeffnet werden". Added a static redirect
to https://axion1337.chat as the final stage.
The prompt stage had 16 unrelated system policies bound (OOBE, user
settings, recovery, etc.), likely from a "select all" slip while
configuring it manually. They crash when evaluated in an anonymous
enrollment context (AnonymousUser has no group_attributes, etc.),
surfacing as opaque errors after form submit. Cleared live and via
the blueprint so re-application doesn't reintroduce them.
The matrix-invitation flow only had Invite+Prompt stage bindings, both
at order=0 (undefined order), missing the Write/Password/Login stages
entirely — invited users were never written to the DB. Applied the fix
live in-cluster (mirroring the working matrix-enrollment stage chain),
and captured it as an Authentik Blueprint (ConfigMap, mounted via
blueprints.configMaps) so the flow state is reproducible via GitOps
instead of manual admin-UI clicks.
Auto-Deploy on Push / verify-and-notify (push) Has been cancelled
- Add upstream_oauth2_config with Authentik provider credentials
- Configure OIDC with client_id and client_secret from Authentik
- Disable local password authentication (OIDC-only login)
- Set claims mapping: subject, localpart, displayname, email
This enables users to login via Authentik OIDC provider with email
and username claims properly mapped for Matrix user provisioning.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The config.json was truncated and had unclosed brackets. This prevented
Helm from properly merging the ElementWeb configuration, so the custom themes
were never loaded into the cluster.
This fix:
- Closes the unclosed JSON brackets
- Validates the full JSON structure
- Removes duplicate/extra closing brackets
- Ensures all 6 custom themes are properly included
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Reduce HelmRelease interval from 1h to 5m for faster sync
- Add checksum annotation to trigger reconciliation when element-values.yaml changes
- This ensures Flux CD re-deploys the chart when themes/config updates are made
To update the checksum after editing element-values.yaml:
md5sum apps/production/custom-configs/element-values.yaml
# Update the value in apps/production/kustomization.yaml patches[0].patch
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Thore CimbalandClaude Sonnet 5
Scrublord MacBad
sorb