docs: document node-level /etc/hosts entry for private-network registry access

Live testing showed pg_dump got "Connection refused" immediately at job
start, but a manual pg_isready/psql from an identically-labeled pod a few
seconds later succeeded fine every time. The cluster's NetworkPolicy
controller needs a brief moment to program a brand-new pod's IP into the
target's allowed ingress rules; a job that starts dumping instantly can
race that window. backup.sh now waits for pg_isready before each pg_dump
(up to 15 tries / ~30s) instead of assuming connectivity is ready at
container start.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

.devcontainer/Dockerfile

@@ -16,11 +16,14 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     zsh \
     sudo \
     openssh-client \
+    gosu \
     && rm -rf /var/lib/apt/lists/*
 
-# Install kubectl
-RUN curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && \
-    echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | tee /etc/apt/sources.list.d/kubernetes.list && \
+# Install kubectl (apt.kubernetes.io was deprecated/shut down by Google in 2023;
+# pkgs.k8s.io is the current community-owned repo, versioned per k8s minor release)
+RUN mkdir -p /etc/apt/keyrings && \
+    curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.34/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg && \
+    echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list && \
     apt-get update && apt-get install -y kubectl && \
     rm -rf /var/lib/apt/lists/*
 
@@ -30,9 +33,10 @@ RUN curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | b
 # Install Flux CLI
 RUN curl -s https://fluxcd.io/install.sh | bash
 
-# Install sops
-RUN SOPS_VERSION=$(curl -s https://api.github.com/repos/getsops/sops/releases/latest | grep tag_name | cut -d '"' -f 4) && \
-    curl -sL -o /usr/local/bin/sops https://github.com/getsops/sops/releases/download/${SOPS_VERSION}/sops-${SOPS_VERSION}.linux.amd64 && \
+# Install sops (arch resolved at build time, same reasoning as the Docker CLI step below)
+RUN SOPS_ARCH=$(dpkg --print-architecture) && \
+    SOPS_VERSION=$(curl -s https://api.github.com/repos/getsops/sops/releases/latest | grep tag_name | cut -d '"' -f 4) && \
+    curl -sL -o /usr/local/bin/sops https://github.com/getsops/sops/releases/download/${SOPS_VERSION}/sops-${SOPS_VERSION}.linux.${SOPS_ARCH} && \
     chmod +x /usr/local/bin/sops
 
 # Install age
@@ -40,17 +44,31 @@ RUN apt-get update && apt-get install -y age && \
     rm -rf /var/lib/apt/lists/*
 
 # Install Docker CLI (for interacting with Docker daemon)
+# arch is resolved at build time so this works on both amd64 (cloud/CI) and arm64 (Apple Silicon) hosts
 RUN curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg && \
-    echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null && \
+    echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null && \
     apt-get update && apt-get install -y docker-ce-cli && \
     rm -rf /var/lib/apt/lists/*
 
 # Create a non-root user 'vscode' for development
-RUN useradd -m -s /bin/bash -G docker vscode && \
+# groupadd is needed because only the Docker CLI (not the daemon) is installed above,
+# so the 'docker' group is never created as a package side effect
+RUN groupadd docker && \
+    useradd -m -s /bin/zsh -G docker vscode && \
     echo "vscode ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers.d/vscode
 
 # Install oh-my-zsh for better shell experience
-RUN su - vscode -c "sh -c '$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)' '' --unattended"
+RUN curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh -o /tmp/install-omz.sh && \
+    su - vscode -c "sh /tmp/install-omz.sh --unattended" && \
+    rm /tmp/install-omz.sh
+
+# Entrypoint runs as root to reconcile the docker group's GID against the mounted
+# socket (see docker-init.sh), then drops to 'vscode' for the actual session/command.
+# Stays root-owned at the PID 1 level; VS Code's own `docker exec -u vscode` sessions
+# and the entrypoint's `gosu vscode` both end up correctly grouped either way.
+COPY docker-init.sh /usr/local/bin/docker-init.sh
+RUN chmod +x /usr/local/bin/docker-init.sh
 
-USER vscode
 WORKDIR /workspace
+ENTRYPOINT ["/usr/local/bin/docker-init.sh"]
+CMD ["/bin/zsh"]

.devcontainer/README.md

@@ -91,7 +91,7 @@ Der Container mounted `~/.age` automatisch. Setze die Umgebungsvariable:
 ```bash
 # Im Container-Terminal (SOPS_AGE_KEY_FILE ist bereits automatisch gesetzt!)
 # Jetzt kannst du Secrets bearbeiten (wird transparent ver-/entschlüsselt):
-sops apps/production/custom-configs/mas-secrets.sops.yaml
+sops apps/production/custom-configs/mas-secret.yaml
 ```
 
 ### Schritt 3: VSCode Integration (optional)
@@ -130,7 +130,7 @@ kubectl get pods -n matrix
 flux get helmreleases -A
 
 # Secrets bearbeiten (mit verschlüsselung)
-sops apps/production/custom-configs/mas-secrets.sops.yaml
+sops apps/production/custom-configs/mas-secret.yaml
 
 # FluxCD Sync erzwingen
 flux reconcile kustomization production-apps --with-source
@@ -193,6 +193,44 @@ Siehe `README.md` → **Issue 3**. Kurz:
 - `wellKnownDelegation: enabled: false` setzen
 - Oder `.well-known/matrix/server` manuell auf `elementWeb` weiterleiten
 
+## ⚠️ Wartungshinweis: Warum dieser Container regelmäßig getestet werden muss
+
+Der Dockerfile installiert mehrere Tools über externe apt-Repos und Install-Skripte
+(`pkgs.k8s.io`, `download.docker.com`, GitHub-Releases, `fluxcd.io`/`ohmyzsh.sh`
+Installer). **Diese Quellen sind nicht unter unserer Kontrolle und können jederzeit
+brechen** — genau das ist am 2026-07-28 passiert: der Container konnte seit
+Fertigstellung nie erfolgreich gebaut werden, ohne dass es jemand bemerkt hat, weil
+niemand ihn zwischenzeitlich tatsächlich gebaut hat. Gefundene und behobene Probleme:
+
+| # | Problem | Ursache | Fix |
+|---|---------|---------|-----|
+| 1 | `apt.kubernetes.io` → `404 Not Found` | Google hat das alte Kubernetes-apt-Repo 2023 abgeschaltet | Umgestellt auf das offizielle Nachfolge-Repo `pkgs.k8s.io` (versioniert pro k8s-Minor-Version, aktuell `v1.34`) |
+| 2 | `docker-ce-cli` "has no installation candidate" auf Apple Silicon | Repo-Zeile hatte `arch=amd64` hartkodiert, Build lief aber auf arm64 | `arch=$(dpkg --print-architecture)` zur Build-Zeit ermitteln |
+| 3 | `useradd: group 'docker' does not exist` | Nur die Docker-**CLI** wird installiert (kein Daemon), daher legt kein Paket die `docker`-Gruppe automatisch an | `groupadd docker` explizit vor `useradd` |
+| 4 | oh-my-zsh-Install schlägt mit Quoting-Fehler fehl | Verschachtelte `sh -c '...'`-Anführungszeichen in einer Zeile | Install-Skript erst in eine Datei laden, dann sauber mit `su - vscode -c "sh /tmp/install-omz.sh --unattended"` ausführen |
+| 5 | `sops`-Binary war hart auf `linux.amd64` gepinnt | Lief auf Apple Silicon nur zufällig per QEMU-Emulation von Docker Desktop mit, nicht nativ | Arch dynamisch über `dpkg --print-architecture` auflösen (`linux.arm64` / `linux.amd64`) |
+| 6 | `docker.sock`-Zugriff im Container: `permission denied` | Der gemountete Host-Socket gehört (je nach Docker-Setup) einer Gruppe/GID, die im Container nicht existiert oder nicht der `docker`-Gruppe entspricht (auf Docker Desktop für Mac/Windows z.B. GID 0/root statt einer eigenen `docker`-Gruppe) | `docker-init.sh`: Root-Entrypoint gleicht beim Container-Start die GID der `docker`-Gruppe an den tatsächlich gemounteten Socket an (bzw. tritt der GID-Inhaber-Gruppe bei, falls die GID schon vergeben ist), wechselt danach per `gosu` zu `vscode` |
+
+**Konsequenz für die Zukunft:** Vor jeder größeren Änderung an `.devcontainer/` (oder
+mindestens vierteljährlich) einmal real bauen und laufen lassen:
+
+```bash
+docker build -f .devcontainer/Dockerfile -t ess-gitops-devcontainer-test .devcontainer
+docker run --rm \
+  -v ~/.kube:/home/vscode/.kube \
+  -v ~/.age:/home/vscode/.age \
+  -v /var/run/docker.sock:/var/run/docker.sock \
+  ess-gitops-devcontainer-test bash -c '
+    kubectl version --client && helm version --short && flux --version && \
+    sops --version && age --version && docker version --format "{{.Server.Version}}" && \
+    id vscode
+  '
+```
+
+Wenn `docker version` hier den echten Server, nicht nur die Client-Version zeigt, und
+`id vscode` die passende Docker-Gruppe/GID auflistet, funktioniert der Socket-Zugriff
+tatsächlich — nicht nur der Build.
+
 ## 📚 Weitere Ressourcen
 
 - [Dev Containers Docs](https://containers.dev)

.devcontainer/docker-init.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+# Runs as root at container start (before any `docker exec -u vscode` from VS Code).
+# The docker.sock's GID is only known once the host socket is actually bind-mounted,
+# so it can't be baked in at image build time - it must be reconciled here, at runtime.
+set -e
+
+if [ -S /var/run/docker.sock ]; then
+  SOCK_GID=$(stat -c '%g' /var/run/docker.sock)
+  CURRENT_GID=$(getent group docker | cut -d: -f3)
+  if [ -n "$SOCK_GID" ] && [ "$SOCK_GID" != "$CURRENT_GID" ]; then
+    EXISTING_GROUP=$(getent group "$SOCK_GID" | cut -d: -f1)
+    if [ -n "$EXISTING_GROUP" ]; then
+      # GID is already taken by another group (e.g. GID 0/root - Docker Desktop for
+      # Mac/Windows owns the socket this way inside its VM), so join that group
+      # instead of trying to reassign it to 'docker'.
+      usermod -aG "$EXISTING_GROUP" vscode
+    else
+      groupmod -g "$SOCK_GID" docker
+    fi
+  fi
+fi
+
+exec gosu vscode "$@"

.devcontainer/postCreateCommand.sh

@@ -26,7 +26,7 @@ echo ""
 echo "📚 Useful commands:"
 echo "  - kubectl get pods -n matrix          (check pod status)"
 echo "  - flux get helmreleases -A            (check helm releases)"
-echo "  - sops apps/production/custom-configs/mas-secrets.sops.yaml  (edit secrets)"
+echo "  - sops apps/production/custom-configs/mas-secret.yaml  (edit secrets)"
 echo ""
 echo "🔗 For kubeconfig setup:"
 echo "  - Copy your ~/.kube/config to access the cluster"

.gitignore

@@ -0,0 +1,2 @@
+.DS_Store
+.claude/

apps/authentik/authentik-backup-secret.yaml

@@ -0,0 +1,23 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: authentik-backup-credentials
+    namespace: authentik
+stringData:
+    borg-passphrase: ENC[AES256_GCM,data:5PApz4TqSNN2vVXeFSuomd051nl+cYk+a+STViwddG/Hj7XWQ099vvTlKSE=,iv:MFG04/66YqtOjZWsLpy236MYwR05z91ngOQ0BmLNxzA=,tag:gxD5prW8Ted3Q2ZY3sYSYQ==,type:str]
+    ssh-private-key: ENC[AES256_GCM,data:87PrkqAIH2xRCu3jlAK3Ts24XKRDoXNCWLomRhizCY43hwLeY5fNzfsGnMTKBt5IxL2bIljx+pbh/Y8IO38ISUUbUAczfi1KeN1gg69wKgqNAlckcmD0dkA2N2F0ZnEFX/rkad68YR8Hw1/A5h3qGUd5FwoVWzIFD/diY86s/LiQBYzwD4bYLUACHqIL7+QyJ6jEsFzA3tLwDySpUpM9OAjFQF0EazObChNYqe/qrc8i1gI8CptWOFkqiZ8Pen1E79gg/zgleOBqSmlcefgZk+JFPAfvORTHkDrHW9X/bLygaTQvskmGNzaKHpe2s/bVafWb7G60ROrpdjYIJfS7c0baRCSrd9kwdM4JQC2gSsMkI4v6oeFSvkTrb684B10zacZdiTKfySDV+Ry4hCZofF2RhAnsTbOpo0XANoOi4InoxUqGPfctVfUCVb6X+HIR5XYxpLrNwWqRpscMP2P3RJsaUZOinmQUKrHX/nMtPZ/ltpwjhQeAXZTtdyPxTCjHAh6fzouuHk66vlCdukXsMQDBdIfdA/3jjGbW,iv:yFNKkdegLLq8jq1Ya6v67urSJTdG3Ge4ZbmKizqQhmk=,tag:6l/qMb1d1oD8sfVEebW6gQ==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBWMllTL2hRcEhCT2MxcWZy
+            dkxndmlZRW5FZ2NiWWdFckxqYTFKS1B6SXlvCjJySXRsZThvczNnWjlyM0N2Snha
+            eitvWUhhdmU4U0V3OG55WjdLbU9KcUEKLS0tIEM3MStzNUJhaGg4M0hKMVF1bnBB
+            RWVqVDRBNmJ0b1Bwd0l3dWxPT3Q0SU0KOEoyejkH4RC0p8ka3FjI7MyzRJg+uu7h
+            j3wf1q+Hgg73djDBSPYJkrB6Bdl4YMwo8SzbtW8O9elDE0qAioR0bQ==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age14l0hwfqylwpemz5y2ghh2yxk0phszlnj3qlejhue0fw0kz3tmfgqdsjzdh
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-07-28T18:20:58Z"
+    mac: ENC[AES256_GCM,data:lwwNbxSxtgDTDmaWMU1uf7TBOEw8gFBFKb982VIsGMeM0fIPHvX12Qts90MNgYIJliOWgAWrwyvAgfXWKuE37RNo+BtyfHCWi4IESKSN/RJrd/yMpRKx+02rifH3nl26ZCAQT1Pa0fjI1SfMhbVzfnD9a/AMARXZMhLRc0OqczY=,iv:SgP2iMtENRtZfw6I9EaOsmvecFNYCIZWqVj+cZ+T7EI=,tag:s4oEyxqyUx6ibqsk4g9xpw==,type:str]
+    version: 3.13.3

apps/authentik/authentik-backup.yaml

@@ -0,0 +1,85 @@
+# Nightly Borg backup of the authentik Postgres database to a Hetzner Storage Box
+# (issues #6 + #15). See apps/authentik/authentik-backup-secret.yaml for the SSH key +
+# Borg repo passphrase, and apps/production/synapse-backup.yaml for the matrix-side job
+# (same Storage Box, separate repo/passphrase, offset schedule).
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: authentik-backup-known-hosts
+  namespace: authentik
+data:
+  known_hosts: |
+    [u641795.your-storagebox.de]:23 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIICf9svRenC/PLKIL9nk6K/pxQgoiFC41wTNvoIncOxs
+---
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: authentik-backup
+  namespace: authentik
+spec:
+  schedule: "15 3 * * *"
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      backoffLimit: 2
+      template:
+        metadata:
+          labels:
+            app.kubernetes.io/name: authentik-backup
+            app.kubernetes.io/component: backup
+        spec:
+          restartPolicy: OnFailure
+          containers:
+            - name: backup
+              image: rohana.axion1337.de/sorb/axion-backup:v2
+              env:
+                - name: BORG_REPO
+                  value: "ssh://u641795@u641795.your-storagebox.de:23/./authentik-backup"
+                - name: BORG_PASSPHRASE
+                  valueFrom:
+                    secretKeyRef:
+                      name: authentik-backup-credentials
+                      key: borg-passphrase
+                - name: SSH_PRIVATE_KEY_FILE
+                  value: /secrets/ssh/ssh-private-key
+                - name: SSH_KNOWN_HOSTS_FILE
+                  value: /secrets/known-hosts/known_hosts
+                - name: DB_HOSTS
+                  value: "authentik:authentik-postgresql"
+                - name: PGUSER
+                  value: authentik
+                - name: PGPASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: authentik-credentials
+                      key: pg-password
+              volumeMounts:
+                - name: ssh-key
+                  mountPath: /secrets/ssh
+                  readOnly: true
+                - name: known-hosts
+                  mountPath: /secrets/known-hosts
+                  readOnly: true
+                - name: scratch
+                  mountPath: /scratch
+              resources:
+                requests:
+                  cpu: 50m
+                  memory: 128Mi
+                limits:
+                  memory: 256Mi
+          volumes:
+            - name: ssh-key
+              secret:
+                secretName: authentik-backup-credentials
+                items:
+                  - key: ssh-private-key
+                    path: ssh-private-key
+                    mode: 0400
+            - name: known-hosts
+              configMap:
+                name: authentik-backup-known-hosts
+            - name: scratch
+              emptyDir: {}

apps/authentik/authentik-blueprints.yaml

@@ -226,3 +226,58 @@ data:
           domain: authentik-default
         attrs:
           default_application: !Find [authentik_core.application, [slug, matrix]]
+  matrix-oidc-provider.yaml: |
+    # yaml-language-server: $schema=https://goauthentik.io/blueprints/schema.json
+    version: 1
+    metadata:
+      name: matrix-oidc-provider
+      labels:
+        blueprints.goauthentik.io/instantiate: "true"
+    entries:
+      # The OIDC Provider + Application linking Authentik to MAS was originally
+      # clicked together by hand in the UI and existed nowhere as code (issue
+      # #36): losing the Authentik DB would have meant re-creating this from
+      # scratch, including a new client_secret that MAS would then no longer
+      # match. client_secret is read from AUTHENTIK_MAS_OIDC_CLIENT_SECRET
+      # (see authentik.yaml HelmRelease values) rather than inlined here,
+      # since this ConfigMap itself is not SOPS-encrypted - the actual value
+      # lives in the authentik-credentials Secret instead.
+      - model: authentik_providers_oauth2.oauth2provider
+        state: present
+        identifiers:
+          name: Matrix Authentication Service
+        id: matrix_mas_provider
+        attrs:
+          client_type: confidential
+          client_id: dHbTAgAgXvjh3VALh220mB3dxcVXAifiXU2ZO3U6
+          client_secret: !Env AUTHENTIK_MAS_OIDC_CLIENT_SECRET
+          # Path includes MAS's own upstream-provider ID, not Authentik's -
+          # must match MAS's config exactly or the OIDC callback breaks.
+          redirect_uris:
+            - matching_mode: strict
+              url: https://account.axion1337.chat/upstream/callback/01KQDJTR1ZVTG8JQ220F5BNBFZ
+          # Stable across username renames - this is what keeps
+          # upstream_oauth_links rows valid after e.g. the elbojoloco rename.
+          sub_mode: hashed_user_id
+          include_claims_in_id_token: true
+          access_code_validity: minutes=1
+          access_token_validity: minutes=5
+          signing_key: !Find [authentik_crypto.certificatekeypair, [name, "authentik Self-signed Certificate"]]
+          authorization_flow: !Find [authentik_flows.flow, [slug, default-provider-authorization-implicit-consent]]
+          invalidation_flow: !Find [authentik_flows.flow, [slug, default-provider-invalidation-flow]]
+          property_mappings:
+            - !Find [authentik_core.propertymapping, [managed, "goauthentik.io/providers/oauth2/scope-openid"]]
+            - !Find [authentik_core.propertymapping, [managed, "goauthentik.io/providers/oauth2/scope-email"]]
+            - !Find [authentik_core.propertymapping, [managed, "goauthentik.io/providers/oauth2/scope-profile"]]
+
+      - model: authentik_core.application
+        state: present
+        identifiers:
+          slug: matrix
+        attrs:
+          name: aXion1337.chat Accountverwaltung
+          provider: !KeyOf matrix_mas_provider
+          meta_description: Matrixclient tailored for aXionCommunity
+          meta_publisher: aXionGaming
+          policy_engine_mode: any
+          open_in_new_tab: false

apps/authentik/authentik-secret.yaml

@@ -7,10 +7,10 @@ stringData:
     secret_key: ENC[AES256_GCM,data:yIyQapbFtFM11LynFtkV3ffExhaDfN9QHeFbI1T0xkIhgsV+9sjg3qwMVmeBlAe7xZl8gsAM4kDj2Q6O91OdDg==,iv:+Cl8vOcxG9/mgRheaCO0bLWyCJXN+f1F2DD3oeHbPFY=,tag:711ytyKf6/tmXomBLoffGA==,type:str]
     pg-password: ENC[AES256_GCM,data:3w8R9mRjMXMJDLjrC8QYaXFHsCU3yYZs2PcaFQNp3Z4=,iv:G/aXgoGz3vBOzZ5K3Y+DDJsqer4F5gvcMmtkzRx93CU=,tag:dXPs1pY/APvnMlxdvB1EkA==,type:str]
     smtp-password: ENC[AES256_GCM,data:JpMgaQFPkBzOg5WjvpmhM0kPwvZkH+4tQjT17RJHjG14WjmWtfG9Bg==,iv:zjQRLIlrxKv5hbd4JZowNUEiibiCUMf79acZY0+dYAc=,tag:ORPafTPyOQJvVvHWQGmqhA==,type:str]
+    mas-oidc-client-secret: ENC[AES256_GCM,data:0yx55FroLSxlnuYgfNwczu3PnbPm1kW74JtiU9oFevVqeQDZc385wU6x5X5TN7owXDO7QaOfGTTMvqIpbwQb6Q5Vt1VMToR+0f44oJcktYoTiDFU9Sy6lR/y6nlvBCNqeJg7vIyVpkIqxwqty15EekyqMpkIMp1fT6Pxmek0SO0=,iv:Ey06ljnqbVARDLVt2sLe8R776VEWpTlzI/+Nka5NocA=,tag:I+GNLHz4V8TFa2ijzK5y2Q==,type:str]
 sops:
     age:
-        - recipient: age14l0hwfqylwpemz5y2ghh2yxk0phszlnj3qlejhue0fw0kz3tmfgqdsjzdh
-          enc: |
+        - enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
             YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRekJuZythYzliTFJ3RlhS
             R2p6TG9NeFdabFlPRWtpNHJMYVVxTWZEcmlRClk0WUorSzdxNlcyWHYwWFBTMnlq
@@ -18,7 +18,8 @@ sops:
             QXVrY1NTeHZkeTlPRWNlVThzWno3T0kKC0KBoLT64GNqb8Ri9u69G7nqb1KftwwP
             /24aVHrPxKi9d4ij9n3bvCYDF4rhtfexhrE4n7CfuKn2DcSiuTniuw==
             -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-04-29T21:43:59Z"
-    mac: ENC[AES256_GCM,data:Y+dJppkaVZ5NOhlvwbbsF5+vDFqGUI1Ps8IcE4J7FIW4HIdMVf6RKM0EInvPUW1LaBlmelCitcE30w0As7ysNRhLY8yUDaKUvuU6mRejlNUIF8wAHzhciL2jTvAQsArHjybJatEig28+wM9VcY8JEa/d/CmuiB9Nq4WbIV+JXlA=,iv:UQj2rIVLNPjtYp3d/jRyNfJyyyUsZ3+NDCgpI4aztzc=,tag:cwiCzG/A+rfRFfLjXVt82w==,type:str]
+          recipient: age14l0hwfqylwpemz5y2ghh2yxk0phszlnj3qlejhue0fw0kz3tmfgqdsjzdh
     encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-07-28T15:54:53Z"
+    mac: ENC[AES256_GCM,data:P6IF+jukwzldK92nHl6s4h6sS4ldXLwpyLpwv2tpI3vFWgTLEnGCnowi2k5lmWUlITEVmLLC0HvsBuduTiGI2sIHHt+r3RdqkV88HGn6oYDVq5a+Ax7ESfqti/4B7ClQCSxl/tU6hBUFe812DiBXJgA03UJQZn8uHY/dP/RgRpc=,iv:V8sqhbJcKglkKsQmJBdgoxDaCYJ3Wt/qRa18jEviH60=,tag:EiNotrYAKIzKndgjU/kTFQ==,type:str]
     version: 3.12.2

apps/authentik/authentik.yaml

@@ -40,6 +40,15 @@ spec:
     global:
       security:
         allowInsecureImages: true
+      # Read by the matrix-oidc-provider blueprint via !Env, so the OAuth2
+      # Provider's client_secret can be captured as code without ever
+      # inlining the live credential into a plain (non-SOPS) ConfigMap.
+      env:
+        - name: AUTHENTIK_MAS_OIDC_CLIENT_SECRET
+          valueFrom:
+            secretKeyRef:
+              name: authentik-credentials
+              key: mas-oidc-client-secret
 
     authentik:
       log_level: info
@@ -89,3 +98,8 @@ spec:
             memory: 256Mi
           limits:
             memory: 512Mi
+        # Chart's own generated policy allows ANY pod in ANY namespace on 5432
+        # (see issue #37) - disabled in favor of our own scoped policy in
+        # apps/authentik/networkpolicy.yaml.
+        networkPolicy:
+          enabled: false

apps/authentik/kustomization.yaml

@@ -9,3 +9,6 @@ resources:
   - authentik.yaml
   - ingress.yaml
   - networkpolicy.yaml
+  # Backup zur Hetzner Storage Box (Issues #6 + #15)
+  - authentik-backup-secret.yaml
+  - authentik-backup.yaml

apps/authentik/networkpolicy.yaml

@@ -4,9 +4,10 @@
 # for upstream OIDC calls. Egress is intentionally untouched (federation-equivalent
 # outbound calls like SMTP aren't restricted here).
 #
-# Note: authentik-postgresql already has its own NetworkPolicy from the Bitnami
-# postgresql subchart (port 5432, no source restriction) - left alone, not duplicated,
-# since it would get reset on the next Helm upgrade anyway.
+# authentik-postgresql: the Bitnami postgresql subchart's own generated NetworkPolicy
+# restricted the port (5432) but not the source - any pod in any namespace could reach
+# it (issue #37). Disabled via postgresql.primary.networkPolicy.enabled: false in
+# authentik.yaml and replaced below with a policy scoped to authentik-server/-worker.
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:
@@ -47,6 +48,35 @@ spec:
 ---
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
+metadata:
+  name: allow-ingress-authentik-postgresql
+  namespace: authentik
+spec:
+  podSelector:
+    matchLabels:
+      app.kubernetes.io/name: postgresql
+      app.kubernetes.io/component: primary
+  policyTypes:
+    - Ingress
+  ingress:
+    - from:
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: authentik
+              app.kubernetes.io/component: server
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: authentik
+              app.kubernetes.io/component: worker
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: authentik-backup
+      ports:
+        - protocol: TCP
+          port: 5432
+---
+apiVersion: networking.k8s.io/v1
+kind: NetworkPolicy
 metadata:
   name: allow-ingress-acme-solver
   namespace: authentik

apps/production/kustomization.yaml

@@ -30,4 +30,7 @@ resources:
   - element-server-suite.yaml
   # Custom Apex Ingress für Element Web + Well-Known auf axion1337.chat
   - apex-ingress.yaml  # Custom Apex Ingress für Element Web + Well-Known auf axion1337.chat
-  - networkpolicy.yaml
+  - networkpolicy.yaml
+  # Backup zur Hetzner Storage Box (Issues #6 + #15)
+  - synapse-backup-secret.yaml
+  - synapse-backup.yaml

apps/production/networkpolicy.yaml

@@ -258,6 +258,9 @@ spec:
         - podSelector:
             matchLabels:
               app.kubernetes.io/name: matrix-authentication-service
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: synapse-backup
       ports:
         - protocol: TCP
           port: 5432

apps/production/synapse-backup-secret.yaml

@@ -0,0 +1,23 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: synapse-backup-credentials
+    namespace: matrix
+stringData:
+    borg-passphrase: ENC[AES256_GCM,data:RRXPwr4UGX30IdozM53abN7ZYztTO3Y1/63dtTh0JSZmU8i8l/ATb4gc3lc=,iv:rIyUr+lOUjo9J53OKZ5ZDmp3d8Nrb9PP2JDK2oCutYU=,tag:MqUmoYgCA03WJQy+RQi04w==,type:str]
+    ssh-private-key: ENC[AES256_GCM,data:320B/lSq7DljCrXZ0BluGv8gLIzYF3KL6VQnR6CM/Vzuf/6qbbZ7lMzrS96XfoubU95OxDeJuOsuNZBQUZNyUGtt4QjQmnl8XHUJzl48xLqh83HLFPtQm2uqU072lscf5OTT7I+JzD2BGks3OIowhrg1q0MdVQdfd4Rhdz8Jphjb+WxvstzNmEX5gxQ9mnBVPmj4DS6ikdXnpe+VDvaCJaVkzD5KgwRmqPy0qbFs9WXziQSo3am5fPeHbwhV3UlRhlok9WrDI40a1T1S5DBhgNhwShq1jAxjr9onuq92REymxAV50oLzsw9ivnH0uimw+3PcplRG1v2xxJ/pimWTCjE17bO7OZ8TyzzwyZ7QA02vSpOMNUdUwVU3N6pSdYQpdTETuVBTVqVc+GKC18Z9fWh2rdwX5eUDTwp0bbDDvPEHdANNxMg3VYD3gwpCgy6/wVnjrO+pVMs8K1CqFn4/H7azhzzeEPkrz27ZBxjlnqvDODsm9tlklr6X4jZDZmTaEmiH+WXuc+1qbNxvsrqtSFfa3CfFGxM1nBOh,iv:sKGsTLsxdQYVUvw7CEARL3YNInSd9LPbFp5Ci5CTgIw=,tag:O1mjNZUsGEqV4uu+LlgD/g==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB3ZUdtN3hyNHlyMkIrbTVS
+            aHQ4OGNUWnA4eTFUVkx4UVp5VWlnMDgxTEU0CkU3M2dMYWgyKytlRVFOVWptZEd2
+            NXlIY0JCd24xcGFzaGpIeks0R2U0U3MKLS0tIHlxYVZ1ZTJsRXNaZ25sVzZtSnp0
+            SEhzQ0tUYzZTRXcwMVNwbG85SHpyb0UKOn3nxy6Y7yQkGargXQ9z6O36vUWW4qJZ
+            D/GbFGmoRi94EtVFdmTGALhjy2D4J9QXy6gHsTapvKyMxF8NEtk+FQ==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age14l0hwfqylwpemz5y2ghh2yxk0phszlnj3qlejhue0fw0kz3tmfgqdsjzdh
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-07-28T18:20:58Z"
+    mac: ENC[AES256_GCM,data:Rur32fQdCyM3nr/X+KeSgmPYEi4nKyh8lqTuSW3TPBVDjwTWMDp1I1ZPPyy5syeW6RHbKputFUzBWVnuQmVrfbZaQ6DBBI5kP9InspUAVUjXDRk9XqiWtdg/wYaTMMJ4Nxv/zdwkh6uJQSG2JHQBWce4NZc2hoPokLR0CjcWiZk=,iv:eWTUj48EFjjtuIIuErMltEdDfabLZeolkpInMYtVP5Q=,tag:6DPxkPqLt0ihJ80WnTzHeA==,type:str]
+    version: 3.13.3

apps/production/synapse-backup.yaml

@@ -0,0 +1,96 @@
+# Nightly Borg backup of the shared Postgres instance (synapse + MAS databases) and the
+# Synapse media_store PVC to a Hetzner Storage Box (issues #6 + #15). See
+# apps/production/synapse-backup-secret.yaml for the SSH key + Borg repo passphrase, and
+# apps/authentik/authentik-backup.yaml for the equivalent authentik-side job.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: synapse-backup-known-hosts
+  namespace: matrix
+data:
+  # Pinned via `ssh-keyscan -p 23 u641795.your-storagebox.de` (2026-07-28) rather than
+  # trusting the host key on first connect in an unattended job.
+  known_hosts: |
+    [u641795.your-storagebox.de]:23 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIICf9svRenC/PLKIL9nk6K/pxQgoiFC41wTNvoIncOxs
+---
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: synapse-backup
+  namespace: matrix
+spec:
+  schedule: "0 3 * * *"
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      backoffLimit: 2
+      template:
+        metadata:
+          labels:
+            app.kubernetes.io/name: synapse-backup
+            app.kubernetes.io/component: backup
+        spec:
+          restartPolicy: OnFailure
+          containers:
+            - name: backup
+              image: rohana.axion1337.de/sorb/axion-backup:v2
+              env:
+                - name: BORG_REPO
+                  value: "ssh://u641795@u641795.your-storagebox.de:23/./synapse-backup"
+                - name: BORG_PASSPHRASE
+                  valueFrom:
+                    secretKeyRef:
+                      name: synapse-backup-credentials
+                      key: borg-passphrase
+                - name: SSH_PRIVATE_KEY_FILE
+                  value: /secrets/ssh/ssh-private-key
+                - name: SSH_KNOWN_HOSTS_FILE
+                  value: /secrets/known-hosts/known_hosts
+                - name: DB_HOSTS
+                  value: "synapse:matrix-stack-postgres matrixauthenticationservice:matrix-stack-postgres"
+                - name: PGUSER
+                  value: postgres
+                - name: PGPASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: matrix-stack-generated
+                      key: POSTGRES_ADMIN_PASSWORD
+                - name: MEDIA_PATH
+                  value: /media/media_store
+              volumeMounts:
+                - name: ssh-key
+                  mountPath: /secrets/ssh
+                  readOnly: true
+                - name: known-hosts
+                  mountPath: /secrets/known-hosts
+                  readOnly: true
+                - name: media
+                  mountPath: /media
+                  readOnly: true
+                - name: scratch
+                  mountPath: /scratch
+              resources:
+                requests:
+                  cpu: 50m
+                  memory: 128Mi
+                limits:
+                  memory: 256Mi
+          volumes:
+            - name: ssh-key
+              secret:
+                secretName: synapse-backup-credentials
+                items:
+                  - key: ssh-private-key
+                    path: ssh-private-key
+                    mode: 0400
+            - name: known-hosts
+              configMap:
+                name: synapse-backup-known-hosts
+            - name: media
+              persistentVolumeClaim:
+                claimName: matrix-stack-synapse-media
+                readOnly: true
+            - name: scratch
+              emptyDir: {}

docs/TASKS.md

@@ -53,6 +53,15 @@ verankert. Was in dieser Session erledigt wurde:
 5. **NetworkPolicies** (Issue #10 ✅ geschlossen) — siehe "Network Security" Abschnitt unten.
    Zwei Live-Incidents beim Rollout (Port-Verwechslungen), beide binnen Minuten live gepatcht
    und danach committed. Nebenbei: `matrixRTC`-Authorisation-Service OOM-Fix (20Mi→128Mi).
+6. **Element Call Qualität nachgeschärft** — 720p-Zwischen-Simulcast-Layer ergänzt (sonst
+   harter Sprung von 1440p auf blockiges 360p bei kleinsten Netzwerkschwankungen), und
+   `video_codec: h264` statt VP8 (klassisches Simulcast wie VP8, kein SVC-Risiko wie bei
+   VP9, oft hardwarebeschleunigt v.a. auf iOS). Live verifiziert: 7/8 Tracks nativ H.264,
+   1 sauberer VP8-Fallback. Deployed als `v0.2.3-elementcall-h264`.
+7. **Backlog nach Gitea migriert** — restlicher offener Backlog (VP9-Retry, ThreadNet-Web-Bug,
+   MAS-Template-Link, WAF und 17 weitere Security-/Infra-Punkte) als Issues #11–#31 angelegt,
+   veraltete erledigte Punkte (Authentik Stage 2/E2E-Test/Invite-Links, Hetzner-Firewall,
+   SSH-Hardening) aus dieser Datei entfernt bzw. als done markiert.
 
 ### Offene Punkte
 - **VP9-Retry**: vermutete Ursache jetzt bekannt (LiveKit nutzt SVC für vp9/av1, Fork-Code
@@ -261,31 +270,22 @@ verankert. Was in dieser Session erledigt wurde:
 - [x] **matrix-recovery Flow gebaut** – war komplett leer (0 Stages); Passwort-Reset funktioniert jetzt, verlinkt von der echten Login-Seite
 - [x] **Brand.default_application gesetzt** – behebt mehrere Dead-Ends, wenn eingeloggte User `/` ohne Ziel aufrufen
 - [x] **2FA/Passkey Selbst-Einrichtung** – Links zu `default-authenticator-totp-setup`/`-webauthn-setup` (2FA bleibt optional, `not_configured_action=skip`), dokumentiert unter `axion1337.chat/docs/setup/security.html`
-- [ ] **Backlog**: MAS Custom-Template-Override, um den 2FA/Passkey-Link direkt auf `account.axion1337.chat/account/` anzuzeigen (statt nur über die Doku-Seite) – erfordert Forken der MAS Tera/Askama-Templates via `templates.path`, höherer Aufwand/Risiko, bei MAS-Updates zu pflegen
+- [ ] **Backlog**: → **Issue #13** (MAS Custom-Template-Override für 2FA/Passkey-Link auf `account.axion1337.chat/account/`)
 
 ---
 
 ## 📋 Backlog (Weitere Aufgaben)
 
-### Authentik Completion
-- [ ] **Finish Authentik Stage 2 – MAS Integration**
-  - Prerequisites: Authentik OIDC Provider vollständig konfiguriert
-  - Task: Update `mas-secret.yaml`, enable password login disable
-  - Commit: `enable-authentik-oidc-integration-in-mas`
-  - Est. Effort: 30 min (manual + scripted)
-
-- [ ] **Test End-to-End Login Flow**
-  - Element Web login → MAS → Authentik → Matrix User Creation
-  - Create test users via Authentik
-  - Verify password reset flow
-  - Commit: (implicit in Stage 2)
-  - Est. Effort: 20 min
-
-- [ ] **Create Invite Links für neue User**
-  - Authentik Admin UI → Invitations → Create
-  - Set expiry dates (7d) + use limits
-  - Document procedure
-  - Est. Effort: 15 min
+**Ab 2026-07-28 in Gitea-Issues gepflegt statt hier** (eine Quelle der Wahrheit) — offene Issues:
+[#6](https://rohana.axion1337.de/sorb/axion1337.chat-gitops/issues/6) DB-Backup-Strategie,
+[#9](https://rohana.axion1337.de/sorb/axion1337.chat-gitops/issues/9) Externe PostgreSQL-Migration,
+[#11](https://rohana.axion1337.de/sorb/axion1337.chat-gitops/issues/11)–[#31](https://rohana.axion1337.de/sorb/axion1337.chat-gitops/issues/31)
+(VP9-Retry, ThreadNet-Web-Build-Bug, MAS-Template-Link, WAF, Media-PVC-Backups, Pod Security
+Admission, Federation-Allowlist, Mjolnir/Draupnir, Content-Scanner, External-Secrets,
+Renovate/Trivy, Security-Advisory-Monitoring, automountServiceAccountToken,
+unattended-upgrades, K3s-API-Security, auditd, Kernel-Hardening, Lynis, CrowdSec, Falco).
+Die detaillierten Beschreibungen unten sind das historische Original, aus dem die Issues
+entstanden sind — nicht mehr getrennt pflegen, stattdessen die Issues aktuell halten.
 
 ### Element Call Enhancement
 - [x] **Element Call Fork für Custom Constraints** (2026-07-28, Closes #8)
@@ -450,24 +450,8 @@ verankert. Was in dieser Session erledigt wurde:
 ## 🔒 Security Hardening (Host & Cluster Level)
 
 ### Host OS Layer (Ubuntu/Debian)
-- [ ] **Hetzner Cloud Firewall**
-  - Default-Deny inbound
-  - Allow: 80/443 (HTTP/HTTPS)
-  - Allow: 22 (SSH) from your IP only (or via WireGuard/Tailscale)
-  - Status: ✅ Can be done in Hetzner UI
-  - Est. Effort: 30 min
-  - Priority: **CRITICAL** (immediate, zero config cost)
-
-- [ ] **SSH Hardening**
-  - Disable password auth (key-only)
-  - Disable root login
-  - PermitRootLogin: no
-  - PasswordAuthentication: no
-  - MaxAuthTries: 3
-  - Optional: Change SSH port (cosmetic, reduces log noise)
-  - Optional: SSH hinter WireGuard/Tailscale (eliminates fail2ban für SSH)
-  - Est. Effort: 2 hours
-  - Priority: **HIGH** (immediate)
+- [x] **Hetzner Cloud Firewall** – Default-Deny inbound, siehe "Phase 7" oben. **Done.**
+- [x] **SSH Hardening** – Key-only, Root-Login disabled, Port 2248, siehe "Phase 7" oben. **Done.**
 
 - [ ] **unattended-upgrades**
   - Enable automatic security updates

docs/install.md

@@ -75,4 +75,59 @@ flux get helmreleases -n matrix --watch
 # Zeigt, wie die Pods hochfahren:
 kubectl get pods -n matrix -w
 ```
-Sobald alle Pods auf `Running` stehen und die Zertifikate über Let's Encrypt validiert wurden (`kubectl get certificate -n matrix`), ist dein Matrix-Stack unter `https://axion1337.chat` erreichbar.
+Sobald alle Pods auf `Running` stehen und die Zertifikate über Let's Encrypt validiert wurden (`kubectl get certificate -n matrix`), ist dein Matrix-Stack unter `https://axion1337.chat` erreichbar.
+
+---
+
+## 🔁 Recovery: lokalen age-Key wiederherstellen (Server läuft bereits)
+
+Anders als Schritt 2 oben (neuen Key **erzeugen**) — falls der Server bereits läuft und nur der
+lokale Rechner den age-Key verloren hat (z.B. nach einer Neuinstallation), lässt sich der
+**bestehende** Private Key direkt aus dem Cluster zurückholen, ohne einen neuen zu generieren
+(das würde `.sops.yaml` und alle bereits verschlüsselten Secrets ungültig machen):
+
+```bash
+mkdir -p ~/.age
+kubectl get secret sops-age -n flux-system -o jsonpath='{.data.age\.agekey}' | base64 -d > ~/.age/keys.txt
+chmod 600 ~/.age/keys.txt
+
+# Public Key zur Kontrolle gegen .sops.yaml abgleichen:
+grep 'public key:' ~/.age/keys.txt
+grep 'age:' .sops.yaml
+```
+
+Voraussetzung: laufender Kubeconfig-Zugriff auf den Cluster (siehe Schritt 1 oben — auch das
+ist reines Zurückkopieren, kein Neu-Erzeugen).
+
+**Bekannte Schwachstelle**: Dieser Key existiert aktuell nur an zwei Orten — im
+`sops-age`-Secret selbst (auf demselben Server) und lokal bei wem auch immer ihn zuletzt
+zurückgeholt hat. Es gibt kein separates, offsite Backup. Fällt der Server komplett aus
+(nicht nur der lokale Rechner), sind alle SOPS-verschlüsselten Secrets im Repo unlesbar.
+Siehe Issue-Backlog für die Entscheidung, ob/wie das abgesichert wird.
+
+---
+
+## 🌐 Node-Konfiguration: `/etc/hosts`-Eintrag für den Gitea-Host
+
+Der K3s-Node und der Gitea-Host (`rohana.axion1337.de`, Container-Registry + Git-Remote)
+teilen sich ein privates Hetzner-Netzwerk (Node `10.0.0.2`, Gitea-Host `10.0.0.3`). Seit
+2026-07-28 hat der Node dafür einen manuellen `/etc/hosts`-Eintrag:
+
+```
+10.0.0.3 rohana.axion1337.de
+```
+
+**Warum**: eine Firewall-Fehlkonfiguration hatte den Node zeitweise komplett von
+`rohana.axion1337.de` über die öffentliche IP abgeschnitten, was Image-Pulls (z.B. für
+Custom-Images wie `sorb/axion-backup`) mit Timeout scheitern ließ. Der Eintrag macht
+Image-Pulls unabhängig vom Zustand der öffentlichen Firewall.
+
+**Wichtig**: Das ist unmanaged Node-Konfiguration (kein GitOps, kein Kubernetes-Objekt) —
+überlebt einen Node-Neuaufbau **nicht** und muss dann erneut gesetzt werden:
+
+```bash
+echo "10.0.0.3 rohana.axion1337.de" | sudo tee -a /etc/hosts
+```
+
+Ein sauberer, clusterweiter Ersatz (z.B. CoreDNS-Rewrite, damit auch Pods selbst intern
+auflösen) ist als Issue #41 nachgehalten.