fix(matrix-stack): shorten HelmRelease reconcile interval to 1m

Live-tested Issue #38's rotation: after merging the automated rotation PR, coturn restarted quickly (Kustomization-level, 1m interval), but synapse-main lagged behind since it depends on this separate HelmRelease with its own 5m interval - a real (self-healing, but avoidable) window where coturn had the new TURN secret and Synapse still had the old one, which would reject each other's credentials/relayed media. Matching the interval to production-apps's 1m tightens that window without needing any new automation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Merge pull request 'chore(coturn): automated TURN shared-secret rotation' (#42 ) from turn-secret-rotation-20260728-192656 into main
2026-07-28 12:00:00 +00:00 · 2026-07-28 12:00:00 +00:00 · 2026-07-28 12:00:00 +00:00 · 2026-07-28 12:00:00 +00:00 · 2026-07-28 12:00:00 +00:00 · 2026-07-28 12:00:00 +00:00
16 changed files with 446 additions and 25 deletions
@@ -0,0 +1,23 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: authentik-backup-credentials
+    namespace: authentik
+stringData:
+    borg-passphrase: ENC[AES256_GCM,data:5PApz4TqSNN2vVXeFSuomd051nl+cYk+a+STViwddG/Hj7XWQ099vvTlKSE=,iv:MFG04/66YqtOjZWsLpy236MYwR05z91ngOQ0BmLNxzA=,tag:gxD5prW8Ted3Q2ZY3sYSYQ==,type:str]
+    ssh-private-key: ENC[AES256_GCM,data:87PrkqAIH2xRCu3jlAK3Ts24XKRDoXNCWLomRhizCY43hwLeY5fNzfsGnMTKBt5IxL2bIljx+pbh/Y8IO38ISUUbUAczfi1KeN1gg69wKgqNAlckcmD0dkA2N2F0ZnEFX/rkad68YR8Hw1/A5h3qGUd5FwoVWzIFD/diY86s/LiQBYzwD4bYLUACHqIL7+QyJ6jEsFzA3tLwDySpUpM9OAjFQF0EazObChNYqe/qrc8i1gI8CptWOFkqiZ8Pen1E79gg/zgleOBqSmlcefgZk+JFPAfvORTHkDrHW9X/bLygaTQvskmGNzaKHpe2s/bVafWb7G60ROrpdjYIJfS7c0baRCSrd9kwdM4JQC2gSsMkI4v6oeFSvkTrb684B10zacZdiTKfySDV+Ry4hCZofF2RhAnsTbOpo0XANoOi4InoxUqGPfctVfUCVb6X+HIR5XYxpLrNwWqRpscMP2P3RJsaUZOinmQUKrHX/nMtPZ/ltpwjhQeAXZTtdyPxTCjHAh6fzouuHk66vlCdukXsMQDBdIfdA/3jjGbW,iv:yFNKkdegLLq8jq1Ya6v67urSJTdG3Ge4ZbmKizqQhmk=,tag:6l/qMb1d1oD8sfVEebW6gQ==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBWMllTL2hRcEhCT2MxcWZy
+            dkxndmlZRW5FZ2NiWWdFckxqYTFKS1B6SXlvCjJySXRsZThvczNnWjlyM0N2Snha
+            eitvWUhhdmU4U0V3OG55WjdLbU9KcUEKLS0tIEM3MStzNUJhaGg4M0hKMVF1bnBB
+            RWVqVDRBNmJ0b1Bwd0l3dWxPT3Q0SU0KOEoyejkH4RC0p8ka3FjI7MyzRJg+uu7h
+            j3wf1q+Hgg73djDBSPYJkrB6Bdl4YMwo8SzbtW8O9elDE0qAioR0bQ==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age14l0hwfqylwpemz5y2ghh2yxk0phszlnj3qlejhue0fw0kz3tmfgqdsjzdh
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-07-28T18:20:58Z"
+    mac: ENC[AES256_GCM,data:lwwNbxSxtgDTDmaWMU1uf7TBOEw8gFBFKb982VIsGMeM0fIPHvX12Qts90MNgYIJliOWgAWrwyvAgfXWKuE37RNo+BtyfHCWi4IESKSN/RJrd/yMpRKx+02rifH3nl26ZCAQT1Pa0fjI1SfMhbVzfnD9a/AMARXZMhLRc0OqczY=,iv:SgP2iMtENRtZfw6I9EaOsmvecFNYCIZWqVj+cZ+T7EI=,tag:s4oEyxqyUx6ibqsk4g9xpw==,type:str]
+    version: 3.13.3
@@ -0,0 +1,85 @@
+# Nightly Borg backup of the authentik Postgres database to a Hetzner Storage Box
+# (issues #6 + #15). See apps/authentik/authentik-backup-secret.yaml for the SSH key +
+# Borg repo passphrase, and apps/production/synapse-backup.yaml for the matrix-side job
+# (same Storage Box, separate repo/passphrase, offset schedule).
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: authentik-backup-known-hosts
+  namespace: authentik
+data:
+  known_hosts: |
+    [u641795.your-storagebox.de]:23 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIICf9svRenC/PLKIL9nk6K/pxQgoiFC41wTNvoIncOxs
+---
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: authentik-backup
+  namespace: authentik
+spec:
+  schedule: "15 3 * * *"
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      backoffLimit: 2
+      template:
+        metadata:
+          labels:
+            app.kubernetes.io/name: authentik-backup
+            app.kubernetes.io/component: backup
+        spec:
+          restartPolicy: OnFailure
+          containers:
+            - name: backup
+              image: rohana.axion1337.de/sorb/axion-backup:v2
+              env:
+                - name: BORG_REPO
+                  value: "ssh://u641795@u641795.your-storagebox.de:23/./authentik-backup"
+                - name: BORG_PASSPHRASE
+                  valueFrom:
+                    secretKeyRef:
+                      name: authentik-backup-credentials
+                      key: borg-passphrase
+                - name: SSH_PRIVATE_KEY_FILE
+                  value: /secrets/ssh/ssh-private-key
+                - name: SSH_KNOWN_HOSTS_FILE
+                  value: /secrets/known-hosts/known_hosts
+                - name: DB_HOSTS
+                  value: "authentik:authentik-postgresql"
+                - name: PGUSER
+                  value: authentik
+                - name: PGPASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: authentik-credentials
+                      key: pg-password
+              volumeMounts:
+                - name: ssh-key
+                  mountPath: /secrets/ssh
+                  readOnly: true
+                - name: known-hosts
+                  mountPath: /secrets/known-hosts
+                  readOnly: true
+                - name: scratch
+                  mountPath: /scratch
+              resources:
+                requests:
+                  cpu: 50m
+                  memory: 128Mi
+                limits:
+                  memory: 256Mi
+          volumes:
+            - name: ssh-key
+              secret:
+                secretName: authentik-backup-credentials
+                items:
+                  - key: ssh-private-key
+                    path: ssh-private-key
+                    mode: 0400
+            - name: known-hosts
+              configMap:
+                name: authentik-backup-known-hosts
+            - name: scratch
+              emptyDir: {}
@@ -9,3 +9,6 @@ resources:
  - authentik.yaml
  - ingress.yaml
  - networkpolicy.yaml
+  # Backup zur Hetzner Storage Box (Issues #6 + #15)
+  - authentik-backup-secret.yaml
+  - authentik-backup.yaml
@@ -68,6 +68,9 @@ spec:
            matchLabels:
              app.kubernetes.io/name: authentik
              app.kubernetes.io/component: worker
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: authentik-backup
      ports:
        - protocol: TCP
          port: 5432
@@ -4,19 +4,28 @@ metadata:
    name: coturn-secret
    namespace: matrix
 stringData:
-    TURN_SECRET: ENC[AES256_GCM,data:SILIqMB+fmAMFITAL7lG1hOgICec6BJf1mOcK0gdmnCHWYqRuJv7jgjfGylG25xzQKi+zE7Qual9PnkZG2KiOA==,iv:+GZqLGusE4Q0x2jEEtFxj06rryyQmQhXdkTy4eE8ZHw=,tag:OpSZkinPTAi1ZKWyo8OX3A==,type:str]
+    TURN_SECRET: ENC[AES256_GCM,data:gab4MSNlRANQz/T/Xn0Z45tkUvQ3Uxn5u0MyYfhZTan49P865UbX4rYPf/KmKsekdwFdOGBzvCgcwxhddVbZ1w==,iv:INbz5UUxgA6xdsPBp5W9Mwe+kd9dhHh2Rx5l3yeNRIQ=,tag:dgk/iFjvst7Wul9Lai2QEQ==,type:str]
 sops:
    age:
-        - recipient: age14l0hwfqylwpemz5y2ghh2yxk0phszlnj3qlejhue0fw0kz3tmfgqdsjzdh
-          enc: |
+        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyRk1mK3NWc1l4T0JCOFpF
-            S0RuQ3ViZmo3QTNVL2JvZ0hzMy91R2l0TEhzCk01a1VGdk1sdVg4aWswTzRibXI4
-            ZlJtNFF5MjBONEZOaWVpeU5taHl2bkEKLS0tIGxpUHY3NUFLWFBaWm1QSlZiVFkx
-            MEJleHFnd3oyT3VPL2dsYkpMUlRkOWMKcKUIgsQ/ff49pGGXMnYwJmwqPVC7woAR
-            IEzvhcNX97xx746SnrxZe5t2YadsYMkYIl0nvqBPJhSlvqMNafpQbQ==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBQMncxOEhRZC9jcHpjT3dW
+            Q1RNaW5pc00zVnJhWHRnZHd3TGhURWNZU2dJCk85bXJkbTEyd1ZybjhDT05pL2c3
+            ZU9EUSt1eDlSQWJyVGtsak1oS3FSR2sKLS0tIEFUdXVHL0V1ZW5VMVVBZEJaYUIw
+            U1BrYlJyQVZkZFhBdmdwbDMyK3lTQkkKEaSy1o+IICf2uaT6olapRJa/duXxjOBg
+            OqRS9axnJ71XxEnHjLTsCbkI5b+8Fux08qKaH9sMsJrWOiSHDdTXXQ==
            -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2026-04-29T21:06:21Z"
-    mac: ENC[AES256_GCM,data:UhyR5m1HYWrZHwNLW5sg2PxbpaydWbP5cekghGlzSpQg7CYEcvZw3tJ/qB8zA19xZSM7tdSHOXdD+QytRq6qW59M1unqMaumA43B6JxQg1C1NdXAW0mkSc2WiNchvgpVii9P/TVlzSSIRwC3YGCQUsfa3SSfNzI4Z6fMuBnhYLE=,iv:4HYxbrYSRJLe1KcQ6q8bpee8/Lx1m3pPmisb/L2Mu64=,tag:l7n3u+Pg6533OzwtNUZvNw==,type:str]
+          recipient: age14l0hwfqylwpemz5y2ghh2yxk0phszlnj3qlejhue0fw0kz3tmfgqdsjzdh
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAwQk82eGdWRmdRNFJQR0xF
+            SWhNRUtNWThZNFM0aXU3V0N4UkhtTy9NTFdnCi9pa3dsVXRja1dTL1pZTnoxQ0JT
+            UUIxekZnVGUvdFgyblFiS0JLMjU3L2cKLS0tIE1IRTJ0M3kvMFZPWVVDYjJlVkk0
+            eGJQOTVUc1NsNE5GdmJtODlmdHp3c2cKHTP6YRMTdYE/iBuSZs/Tjt4TwKCxHEIu
+            f3jTblKIqWwRHKCgOIkC16QDbpMBlNLH3JknJEdIjkB2HIrXrw1MNA==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1x4jjwc8nuttwr8us924pvdc6dll5npkc6c8f4zf2hx5d2qu75dtqx0fm0d
    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-07-28T19:26:57Z"
+    mac: ENC[AES256_GCM,data:zU3i1WGY1X4igBYzfC1WAWdtMQZWGBSacaJpXnUVt6+BCfucqaH807IRWpWbTR0+xuLteaqcdXuVuAlStqs/VV586Q4cD2FfWDeBq/O09HEwTrfekTjAyyu7zCFyZ2ja+kB4DQ83NoscCiNwNmfTZF+6RzMqx7yphPxK5go1r7c=,iv:6FIYWsNvz78zJscbxyDgy1u6ur20MnOMkMcsA4329OI=,tag:uSDpAWEixGcVmtOloMoKuw==,type:str]
    version: 3.12.2
@@ -65,6 +65,17 @@ metadata:
  namespace: matrix
 spec:
  replicas: 1
+  # hostNetwork pods bind directly to the node's ports (3478/5349) - on this single-node
+  # cluster, RollingUpdate's default "bring up the new pod before removing the old one"
+  # can never schedule (port conflict). Recreate kills the old pod first.
+  # Note: switching to Recreate on an existing Deployment that already had the
+  # RollingUpdate defaults recorded required a one-time manual
+  # `kubectl patch --type=merge -p '{"spec":{"strategy":{"rollingUpdate":null,"type":"Recreate"}}}'`
+  # (2026-07-28) - a YAML `rollingUpdate: null` in this file is dropped before reaching the
+  # API server (client-side omits null keys) rather than sent as an explicit field deletion,
+  # so it can't clear an already-set field on its own.
+  strategy:
+    type: Recreate
  selector:
    matchLabels:
      app: coturn
@@ -74,6 +85,10 @@ spec:
        app: coturn
      annotations:
        prometheus.io/scrape: "false"
+        # Bumped on every TURN_SECRET rotation (Issue #38) to force a new pod, since
+        # Kubernetes doesn't restart running pods when a referenced Secret's content
+        # changes and the initContainer that reads it only runs once at pod start.
+        rotated-at: "2026-07-28T19:26:57Z"
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
@@ -58,15 +58,6 @@ data:
            room_list_publication_rules:
              - user_id: "*"
                action: allow
-        turn:
-          config: |
-            turn_uris:
-              - "turn:turn.axion1337.chat?transport=udp"
-              - "turn:turn.axion1337.chat?transport=tcp"
-              - "turns:turn.axion1337.chat?transport=tcp"
-            turn_shared_secret: "cab3c8408363515d9b4cdc3384a1f76ca17a973242fdfdc72b67ac4d86158527"
-            turn_user_lifetime: 86400000
-            turn_allow_guests: false
        oembed:
          config: |
            oembed_enabled: true
@@ -4,7 +4,11 @@ metadata:
  name: matrix-stack
  namespace: matrix
 spec:
-  interval: 5m
+  # Shortened from 5m to match production-apps Kustomization's 1m interval - narrows the
+  # window between coturn (Kustomization-only, no Helm indirection) and synapse-main
+  # (behind this HelmRelease) picking up a rotated TURN secret after Issue #38's
+  # automated-rotation PR gets merged. Self-heals either way, just faster now.
+  interval: 1m
  chart:
    spec:
      chart: matrix-stack
@@ -25,7 +29,10 @@ spec:
    - kind: Secret
      name: ess-mas-values-secret
      valuesKey: values.yaml
-      
+    - kind: Secret
+      name: synapse-turn-secret
+      valuesKey: values.yaml
+
  values:
    # Top-Level: serverName – das ist dein Matrix-Homeserver-Name
    serverName: axion1337.chat
@@ -1,8 +1,10 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization

-# Patch: Fügt einen Checksum der element-values.yaml zur HelmRelease hinzu
-# Damit wird Flux die HelmRelease neu-synced wenn sich die ConfigMap ändert
+# Patch: Fügt Checksums der element-values.yaml und des turn_shared_secret zur
+# HelmRelease hinzu. Damit wird Flux die HelmRelease neu-synced (und synapse-main neu
+# gestartet), wenn sich die jeweilige ConfigMap/Secret ändert - siehe Issue #38's
+# Rotations-Mechanismus, der turn-secret-checksum bei jeder Rotation bumpt.
 patches:
  - target:
      kind: HelmRelease
@@ -12,6 +14,9 @@ patches:
      - op: add
        path: /metadata/annotations/element-config-checksum
        value: "401f8a87d0ef5d91d2e5032d4aede42c"
+      - op: add
+        path: /metadata/annotations/turn-secret-checksum
+        value: "d220c0e4ff8f7106328c8827d47e8734"

 resources:
  - matrix-postgres-auth.yaml
@@ -26,8 +31,15 @@ resources:
  # TURN Server für WebRTC
  - coturn-secret.yaml
  - coturn.yaml
+  - synapse-turn-secret.yaml
  # HelmRelease (muss ganz unten stehen, damit die ConfigMaps vorher da sind!)
  - element-server-suite.yaml
  # Custom Apex Ingress für Element Web + Well-Known auf axion1337.chat
  - apex-ingress.yaml  # Custom Apex Ingress für Element Web + Well-Known auf axion1337.chat
-  - networkpolicy.yaml
+  - networkpolicy.yaml
+  # Backup zur Hetzner Storage Box (Issues #6 + #15)
+  - synapse-backup-secret.yaml
+  - synapse-backup.yaml
+  # Automatisierte TURN-Secret-Rotation (Issue #38)
+  - turn-secret-rotation-secret.yaml
+  - turn-secret-rotation.yaml
@@ -258,6 +258,9 @@ spec:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: matrix-authentication-service
+        - podSelector:
+            matchLabels:
+              app.kubernetes.io/name: synapse-backup
      ports:
        - protocol: TCP
          port: 5432
@@ -0,0 +1,23 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: synapse-backup-credentials
+    namespace: matrix
+stringData:
+    borg-passphrase: ENC[AES256_GCM,data:RRXPwr4UGX30IdozM53abN7ZYztTO3Y1/63dtTh0JSZmU8i8l/ATb4gc3lc=,iv:rIyUr+lOUjo9J53OKZ5ZDmp3d8Nrb9PP2JDK2oCutYU=,tag:MqUmoYgCA03WJQy+RQi04w==,type:str]
+    ssh-private-key: ENC[AES256_GCM,data:320B/lSq7DljCrXZ0BluGv8gLIzYF3KL6VQnR6CM/Vzuf/6qbbZ7lMzrS96XfoubU95OxDeJuOsuNZBQUZNyUGtt4QjQmnl8XHUJzl48xLqh83HLFPtQm2uqU072lscf5OTT7I+JzD2BGks3OIowhrg1q0MdVQdfd4Rhdz8Jphjb+WxvstzNmEX5gxQ9mnBVPmj4DS6ikdXnpe+VDvaCJaVkzD5KgwRmqPy0qbFs9WXziQSo3am5fPeHbwhV3UlRhlok9WrDI40a1T1S5DBhgNhwShq1jAxjr9onuq92REymxAV50oLzsw9ivnH0uimw+3PcplRG1v2xxJ/pimWTCjE17bO7OZ8TyzzwyZ7QA02vSpOMNUdUwVU3N6pSdYQpdTETuVBTVqVc+GKC18Z9fWh2rdwX5eUDTwp0bbDDvPEHdANNxMg3VYD3gwpCgy6/wVnjrO+pVMs8K1CqFn4/H7azhzzeEPkrz27ZBxjlnqvDODsm9tlklr6X4jZDZmTaEmiH+WXuc+1qbNxvsrqtSFfa3CfFGxM1nBOh,iv:sKGsTLsxdQYVUvw7CEARL3YNInSd9LPbFp5Ci5CTgIw=,tag:O1mjNZUsGEqV4uu+LlgD/g==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB3ZUdtN3hyNHlyMkIrbTVS
+            aHQ4OGNUWnA4eTFUVkx4UVp5VWlnMDgxTEU0CkU3M2dMYWgyKytlRVFOVWptZEd2
+            NXlIY0JCd24xcGFzaGpIeks0R2U0U3MKLS0tIHlxYVZ1ZTJsRXNaZ25sVzZtSnp0
+            SEhzQ0tUYzZTRXcwMVNwbG85SHpyb0UKOn3nxy6Y7yQkGargXQ9z6O36vUWW4qJZ
+            D/GbFGmoRi94EtVFdmTGALhjy2D4J9QXy6gHsTapvKyMxF8NEtk+FQ==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age14l0hwfqylwpemz5y2ghh2yxk0phszlnj3qlejhue0fw0kz3tmfgqdsjzdh
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-07-28T18:20:58Z"
+    mac: ENC[AES256_GCM,data:Rur32fQdCyM3nr/X+KeSgmPYEi4nKyh8lqTuSW3TPBVDjwTWMDp1I1ZPPyy5syeW6RHbKputFUzBWVnuQmVrfbZaQ6DBBI5kP9InspUAVUjXDRk9XqiWtdg/wYaTMMJ4Nxv/zdwkh6uJQSG2JHQBWce4NZc2hoPokLR0CjcWiZk=,iv:eWTUj48EFjjtuIIuErMltEdDfabLZeolkpInMYtVP5Q=,tag:6DPxkPqLt0ihJ80WnTzHeA==,type:str]
+    version: 3.13.3
@@ -0,0 +1,96 @@
+# Nightly Borg backup of the shared Postgres instance (synapse + MAS databases) and the
+# Synapse media_store PVC to a Hetzner Storage Box (issues #6 + #15). See
+# apps/production/synapse-backup-secret.yaml for the SSH key + Borg repo passphrase, and
+# apps/authentik/authentik-backup.yaml for the equivalent authentik-side job.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: synapse-backup-known-hosts
+  namespace: matrix
+data:
+  # Pinned via `ssh-keyscan -p 23 u641795.your-storagebox.de` (2026-07-28) rather than
+  # trusting the host key on first connect in an unattended job.
+  known_hosts: |
+    [u641795.your-storagebox.de]:23 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIICf9svRenC/PLKIL9nk6K/pxQgoiFC41wTNvoIncOxs
+---
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: synapse-backup
+  namespace: matrix
+spec:
+  schedule: "0 3 * * *"
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      backoffLimit: 2
+      template:
+        metadata:
+          labels:
+            app.kubernetes.io/name: synapse-backup
+            app.kubernetes.io/component: backup
+        spec:
+          restartPolicy: OnFailure
+          containers:
+            - name: backup
+              image: rohana.axion1337.de/sorb/axion-backup:v2
+              env:
+                - name: BORG_REPO
+                  value: "ssh://u641795@u641795.your-storagebox.de:23/./synapse-backup"
+                - name: BORG_PASSPHRASE
+                  valueFrom:
+                    secretKeyRef:
+                      name: synapse-backup-credentials
+                      key: borg-passphrase
+                - name: SSH_PRIVATE_KEY_FILE
+                  value: /secrets/ssh/ssh-private-key
+                - name: SSH_KNOWN_HOSTS_FILE
+                  value: /secrets/known-hosts/known_hosts
+                - name: DB_HOSTS
+                  value: "synapse:matrix-stack-postgres matrixauthenticationservice:matrix-stack-postgres"
+                - name: PGUSER
+                  value: postgres
+                - name: PGPASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: matrix-stack-generated
+                      key: POSTGRES_ADMIN_PASSWORD
+                - name: MEDIA_PATH
+                  value: /media/media_store
+              volumeMounts:
+                - name: ssh-key
+                  mountPath: /secrets/ssh
+                  readOnly: true
+                - name: known-hosts
+                  mountPath: /secrets/known-hosts
+                  readOnly: true
+                - name: media
+                  mountPath: /media
+                  readOnly: true
+                - name: scratch
+                  mountPath: /scratch
+              resources:
+                requests:
+                  cpu: 50m
+                  memory: 128Mi
+                limits:
+                  memory: 256Mi
+          volumes:
+            - name: ssh-key
+              secret:
+                secretName: synapse-backup-credentials
+                items:
+                  - key: ssh-private-key
+                    path: ssh-private-key
+                    mode: 0400
+            - name: known-hosts
+              configMap:
+                name: synapse-backup-known-hosts
+            - name: media
+              persistentVolumeClaim:
+                claimName: matrix-stack-synapse-media
+                readOnly: true
+            - name: scratch
+              emptyDir: {}
@@ -0,0 +1,31 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: synapse-turn-secret
+    namespace: matrix
+stringData:
+    values.yaml: ENC[AES256_GCM,data:An+GypE/btGmfnvL1rPzvbN1sAnZKQnHV20GxXkZiRb+Wi8BymAUA6JRBo5mdtvK1dgDMF6H1wN+Y7wjzwduteaoskVfgd7Xw+lL1SrZ/6pchGxlUIWSJlLTmp6IHZaruleDUsCKdrWfN9TQDE+WYdiECBktlGGXs8ZZMTzjHmHKJ3tNwSimaxuG662Amb0LiHLX+zBUKOuG0L4cWrp6JI3NKQFgNNiF6xwXjHDX550bGOQ2pi5UVPL9+HlH2tPeTzoCCZdJLWF+2kZg31bZ7dzRp8M23eDrzfGC9e9m0NrA2z1ljagbnuT0tixp6J8205qfBOIOVOWbXHmd/LQ8nphA64En6JXqU8XeZ+mgMUmFuGwyewLq4oilLoQfdZujEyGhc9Hd/yonRlc/RDBlONlI8d+IWqZxf0AaUYEAe9uE6MulqRs0CowPUpCmWtZjz68qQlIvJEAqxNdWEnp+SkExDNNe9y8tBOCXy726TRX0MLIlZ+CjPDCHhwDeKw5oQU0wXr9QezY=,iv:bPK1+PViBg046N3q2Eb1ZOJOYv7X/HgpCEU/MLp7o6k=,tag:/2Kyv7aB4baOL+3KdAeQ9w==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBuK2xIT3M0cGtVOHdKYklF
+            OHl1VHFRTkVSaHVQNUUyRlhZUjRWM3JUN0hBCnpUS2YzdG1mSjlRaEVvTHdKVkR4
+            L1hrR05IRUdqdGp2aU95aWxRRXdsQWcKLS0tICtLbnFsK3Zza053VXFWdVErRy92
+            WUUzZTBIRzUyWnp5a1ZScUVqb0NyencKvnFyJCR6j1/aH4gJvFmLPNlk5XpC08wF
+            mTmL981uGfz4NULc+O3sDkonJ827glpefgWPgPW2HmKT88d4A9vyJw==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age14l0hwfqylwpemz5y2ghh2yxk0phszlnj3qlejhue0fw0kz3tmfgqdsjzdh
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBVY0NXa1B5c2RzbzRReU1h
+            RHBSbk9aK3d6VjRWZ1hzL3FDZFZJZGFqTVQ0Ck9ib0p3bkR6cU8yc0VFNjEzSFFi
+            T3dWY1luQW1VZ1NjZFNoZFFLSCsvelkKLS0tIHIxV0d6TWhpSEc3d3c0L1VvOWxk
+            eDhTM2pDbTNXZXlWRVYxR2tPQU5iLzgKrLLLSBU/g5ebeRNi7hWYbcuJ/2JOfiUn
+            0DBnzMVJPBfqq/u8THiRYaMajx3k4D9+FN7qc5nBgTd85iGBo+OowA==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1x4jjwc8nuttwr8us924pvdc6dll5npkc6c8f4zf2hx5d2qu75dtqx0fm0d
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-07-28T19:26:57Z"
+    mac: ENC[AES256_GCM,data:UfD+QJel9FygoKPM02KdOLvP1id/21ghQ0uWAQJxEs/w+B2vVkpe/dXSJ7QnI0e528ns1HW81GAxkxyAupsUDq8/pBdEQgItwDs5DIgHvH+sTPAtj2Kyq2wMOeEt+XDGLRdOlAwbcKDS/LyfeZ5b9QnAbY2w/tTinTkzEvbudT8=,iv:WmmwjAmZmM0W8dvqBcv9XYZkYtan3lg0Sb8ypVuGoJk=,tag:3yzZhSFtbSbLUXSdDWGAtw==,type:str]
+    version: 3.13.3
@@ -0,0 +1,23 @@
+apiVersion: v1
+kind: Secret
+metadata:
+    name: turn-secret-rotation-credentials
+    namespace: matrix
+stringData:
+    age-key.txt: ENC[AES256_GCM,data:4LAs9LLFo38UMHXCo4lun9RHxGnDyp7GWlaNdIqqkSL9lNv7+ILdlc03CxFVobCYxK65xMOn1xdEty+887JBMlawST04am/5MkAnUivKwXCw8OHmbZhCwKHFqSYH/NsgVNf+btZKIIny8XPVQAPj/vQIi+Ity+BQyPkEZ1WUcsqjDoaK9IFhQTePJtHgWivhOY2WpUt/TP7vTfub4TOrgVpzNd9LIpBkwq+zhpVXKwnYUWOuLMXlPe08kazy,iv:9A24HbTl24slj+qTCfyI01+dGqRFVPDUA0wp2kSUHpc=,tag:iQxU0dWD4Noo1m0HXhV6vQ==,type:str]
+    gitea-token: ENC[AES256_GCM,data:cmZ1GCrqRYLtLn+cRVZCrO7UcCIavlQLJPt2PRMtBbgLdhDVTElUKA==,iv:9v29GXHRtDlrL3PoRCdOqYpBepZrX04+6UjoywRZX0E=,tag:ZvYXBOm9qgB1XA5lkV4LsQ==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA5S3BtTHNLVGVVM2ZpTlZS
+            Y3pGMS9CSkNsdUpPbWtkSTRHK0p6U2lwdEJnClVWdXp4SllyM1hvbTZyTU40SDc2
+            QlVtMDduZWpaVENiYnhMNlFXd01QblEKLS0tIGNzTGRZcmoyaFltUHRDSHBPZE1N
+            OCszUkl1VjQ5V3F2cVI4dXJFcER5YXcK+2Eh1JNLuMiCnpQ3cL/I7XTykkIZ3tqp
+            O3c9UwYs1FAZWlMgElTBTqsmut1ShduIYfDFRKGeS0UxPEM4U+tIGw==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age14l0hwfqylwpemz5y2ghh2yxk0phszlnj3qlejhue0fw0kz3tmfgqdsjzdh
+    encrypted_regex: ^(data|stringData)$
+    lastmodified: "2026-07-28T19:21:28Z"
+    mac: ENC[AES256_GCM,data:vx6Bs/L0NXKUvvQdu6aYtuur/CYPkIBZzvFLqTyd08Errw0dGMrg73oHQ/imxpe42HgnO2mGwxNdEx2jYYbtc3RBWHE/yPH5m8y/XLoSL3fauzbkGsDwMSWzKiZXyIuGh7SxuB+CFY9qqFMK+dap1Ofno7a1/Gr1qibVDqscwxw=,iv:9AUQQuTCja09OohzVw73URMHE8xCW7iLLtBg7GSDcPA=,tag:g1ibnBIGqokvs2IEVOYq6Q==,type:str]
+    version: 3.13.3
@@ -0,0 +1,70 @@
+# Automated TURN shared-secret rotation (Issue #38). Generates a new secret, re-encrypts
+# apps/production/coturn-secret.yaml and synapse-turn-secret.yaml using a dedicated,
+# narrowly-scoped age key (see turn-secret-rotation-secret.yaml - it can only decrypt these
+# two files, not the repo's master sops-age key), bumps the checksum/rotated-at annotations
+# so a merge restarts both consumers automatically, and opens a Pull Request rather than
+# pushing straight to main - a human reviews and merges it.
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: turn-secret-rotation
+  namespace: matrix
+spec:
+  schedule: "0 4 1 * *"
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      backoffLimit: 1
+      template:
+        metadata:
+          labels:
+            app.kubernetes.io/name: turn-secret-rotation
+            app.kubernetes.io/component: rotation
+        spec:
+          restartPolicy: OnFailure
+          # Public-internet reachability to the Gitea host has been flaky (see Issue #41);
+          # both servers share a private Hetzner network. hostAliases (unlike the node-level
+          # /etc/hosts fix used for image pulls) is actually honored by in-pod processes.
+          hostAliases:
+            - ip: "10.0.0.3"
+              hostnames:
+                - "rohana.axion1337.de"
+          containers:
+            - name: rotate
+              image: rohana.axion1337.de/sorb/axion-secret-rotation:v1
+              env:
+                - name: GITEA_HOST
+                  value: "rohana.axion1337.de"
+                - name: GITEA_REPO
+                  value: "sorb/axion1337.chat-gitops"
+                - name: GITEA_TOKEN
+                  valueFrom:
+                    secretKeyRef:
+                      name: turn-secret-rotation-credentials
+                      key: gitea-token
+                - name: SOPS_AGE_KEY_FILE
+                  value: /secrets/age/age-key.txt
+                - name: GIT_AUTHOR_NAME
+                  value: "turn-secret-rotation"
+                - name: GIT_AUTHOR_EMAIL
+                  value: "turn-secret-rotation@axion1337.chat"
+              volumeMounts:
+                - name: age-key
+                  mountPath: /secrets/age
+                  readOnly: true
+              resources:
+                requests:
+                  cpu: 50m
+                  memory: 64Mi
+                limits:
+                  memory: 128Mi
+          volumes:
+            - name: age-key
+              secret:
+                secretName: turn-secret-rotation-credentials
+                items:
+                  - key: age-key.txt
+                    path: age-key.txt
+                    mode: 0400
@@ -103,4 +103,31 @@ ist reines Zurückkopieren, kein Neu-Erzeugen).
 `sops-age`-Secret selbst (auf demselben Server) und lokal bei wem auch immer ihn zuletzt
 zurückgeholt hat. Es gibt kein separates, offsite Backup. Fällt der Server komplett aus
 (nicht nur der lokale Rechner), sind alle SOPS-verschlüsselten Secrets im Repo unlesbar.
-Siehe Issue-Backlog für die Entscheidung, ob/wie das abgesichert wird.
+Siehe Issue-Backlog für die Entscheidung, ob/wie das abgesichert wird.
+
+---
+
+## 🌐 Node-Konfiguration: `/etc/hosts`-Eintrag für den Gitea-Host
+
+Der K3s-Node und der Gitea-Host (`rohana.axion1337.de`, Container-Registry + Git-Remote)
+teilen sich ein privates Hetzner-Netzwerk (Node `10.0.0.2`, Gitea-Host `10.0.0.3`). Seit
+2026-07-28 hat der Node dafür einen manuellen `/etc/hosts`-Eintrag:
+
+```
+10.0.0.3 rohana.axion1337.de
+```
+
+**Warum**: eine Firewall-Fehlkonfiguration hatte den Node zeitweise komplett von
+`rohana.axion1337.de` über die öffentliche IP abgeschnitten, was Image-Pulls (z.B. für
+Custom-Images wie `sorb/axion-backup`) mit Timeout scheitern ließ. Der Eintrag macht
+Image-Pulls unabhängig vom Zustand der öffentlichen Firewall.
+
+**Wichtig**: Das ist unmanaged Node-Konfiguration (kein GitOps, kein Kubernetes-Objekt) —
+überlebt einen Node-Neuaufbau **nicht** und muss dann erneut gesetzt werden:
+
+```bash
+echo "10.0.0.3 rohana.axion1337.de" | sudo tee -a /etc/hosts
+```
+
+Ein sauberer, clusterweiter Ersatz (z.B. CoreDNS-Rewrite, damit auch Pods selbst intern
+auflösen) ist als Issue #41 nachgehalten.
Author	SHA1	Message	Date
Thore CimbalandClaude Sonnet 5	2932ffffc3	fix(matrix-stack): shorten HelmRelease reconcile interval to 1m Live-tested Issue #38's rotation: after merging the automated rotation PR, coturn restarted quickly (Kustomization-level, 1m interval), but synapse-main lagged behind since it depends on this separate HelmRelease with its own 5m interval - a real (self-healing, but avoidable) window where coturn had the new TURN secret and Synapse still had the old one, which would reject each other's credentials/relayed media. Matching the interval to production-apps's 1m tightens that window without needing any new automation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-28 12:00:00 +00:00
Thore Cimbal	0f7ff1a896	Merge pull request 'chore(coturn): automated TURN shared-secret rotation' (#42 ) from turn-secret-rotation-20260728-192656 into main	2026-07-28 12:00:00 +00:00
Thore CimbalandClaude Sonnet 5	debf980d5f	fix(turn-rotation): add hostAliases for private-network Gitea access Job hung indefinitely on git clone - public-internet reachability to rohana.axion1337.de from inside pods is currently down (related to Issue #41's firewall/private-network findings), and the node-level /etc/hosts workaround only helps containerd's image pulls, not processes running inside pods (they resolve via CoreDNS + their own /etc/hosts, not the node's). hostAliases routes this job's git/API traffic over the private Hetzner network directly - verified working with a real pod. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-28 12:00:00 +00:00
turn-secret-rotation	802e09cd9c	chore(coturn): automated TURN shared-secret rotation Scheduled rotation (Issue #38). New secret generated, re-encrypted with the scoped rotation age key, checksum/rotated-at annotations bumped so Flux restarts coturn + synapse-main on merge. Please review and merge.	2026-07-28 12:00:00 +00:00
Thore Cimbal	d742c99ba5	chore: set Gitea PAT for TURN secret rotation job	2026-07-28 12:00:00 +00:00
Thore CimbalandClaude Sonnet 5	ac7e65100d	feat(coturn): add automated TURN shared-secret rotation CronJob Closes issue #38's automation half (architecture fix + first rotation already landed in earlier commits this session). Monthly CronJob (rohana.axion1337.de/sorb/axion-secret-rotation:v1 - alpine + git/sops/ jq/age) that: - generates a new secret - re-encrypts coturn-secret.yaml and synapse-turn-secret.yaml using the scoped rotation-only age key (added as an additional recipient in an earlier commit) - never touches the repo's master sops-age key - bumps the turn-secret-checksum (HelmRelease annotation) and rotated-at (coturn Deployment annotation) so merging actually restarts both consumers, reusing the existing checksum-annotation pattern already in this repo rather than inventing a new mechanism - opens a Pull Request rather than pushing straight to main - a human reviews and merges, keeping a checkpoint before production picks up new credentials while still automating the tedious coordination work Needs a Gitea PAT (repo write scope) filled into turn-secret-rotation-secret.yaml's gitea-token key before first use. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-28 12:00:00 +00:00
Thore CimbalandClaude Sonnet 5	be95bd921c	feat(coturn): add scoped age recipient for automated rotation job Adds a second age recipient (dedicated keypair, private key never touches the repo) to just these two files, so the upcoming rotation CronJob can decrypt/re-encrypt them without needing the repo's master sops-age key, which can decrypt every secret in the repo. Verified: master key still decrypts both files; the new scoped key can decrypt both files but correctly cannot decrypt an unrelated secret (authentik-secret.yaml). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-28 12:00:00 +00:00
Thore Cimbal	7795f89fc9	docs(coturn): note the one-time manual rollingUpdate fix instead of an ineffective YAML null	2026-07-28 12:00:00 +00:00
Thore CimbalandClaude Sonnet 5	9fa947cc21	fix(coturn): explicitly null rollingUpdate field alongside Recreate strategy Server-side apply left the old RollingUpdate defaults (maxSurge/maxUnavailable) on the live object from before strategy.type was ever set explicitly, which the API rejects when combined with type: Recreate. Explicit null clears it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-28 12:00:00 +00:00
Thore CimbalandClaude Sonnet 5	233b13f972	fix(coturn): use Recreate deployment strategy for hostNetwork pod Discovered live: the default RollingUpdate strategy tries to schedule the new pod before removing the old one, which can never succeed for a hostNetwork Deployment on a single-node cluster (both pods would need the same node ports 3478/5349 simultaneously). Surfaced by the rotated-at annotation bump from the previous commit getting stuck in FailedScheduling. Recreate kills the old pod first, freeing the ports. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-28 12:00:00 +00:00
Thore CimbalandClaude Sonnet 5	51ea51352b	fix(coturn): remove plaintext TURN shared secret, rotate leaked value Issue #38 discussion surfaced a real bug: the TURN shared secret was duplicated - correctly SOPS-encrypted in coturn-secret.yaml, but also hardcoded in plaintext in synapse-values.yaml (a plain, non-SOPS ConfigMap), visible in git history. Also found turn_user_lifetime is 86400000ms (24h), not "short-lived" as previously assumed - raising the stakes of the leak somewhat. Extracted the turn config block into its own dedicated SOPS-encrypted Secret (synapse-turn-secret.yaml), wired via a second HelmRelease valuesFrom entry (same pattern already used for ess-mas-values-secret). Rotated the value while doing this, so the leaked plaintext secret is no longer live anywhere. Added checksum/rotated-at annotations (matrix-stack HelmRelease's existing element-config-checksum patch gets a sibling turn-secret-checksum; coturn's Deployment pod template gets a rotated-at annotation) so future rotations actually restart both consumers - Kubernetes doesn't restart running pods when a referenced Secret's content changes on its own. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-28 12:00:00 +00:00
Thore Cimbal	9607687e4e	docs: document node-level /etc/hosts entry for private-network registry access	2026-07-28 12:00:00 +00:00
Thore CimbalandClaude Sonnet 5	49005ed325	fix(backup): retry pg_dump connection to survive NetworkPolicy sync lag Live testing showed pg_dump got "Connection refused" immediately at job start, but a manual pg_isready/psql from an identically-labeled pod a few seconds later succeeded fine every time. The cluster's NetworkPolicy controller needs a brief moment to program a brand-new pod's IP into the target's allowed ingress rules; a job that starts dumping instantly can race that window. backup.sh now waits for pg_isready before each pg_dump (up to 15 tries / ~30s) instead of assuming connectivity is ready at container start. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-28 12:00:00 +00:00
Thore CimbalandClaude Sonnet 5	ba270bce71	feat(backup): add nightly Borg backups to Hetzner Storage Box Closes issues #6 and #15 (treated jointly, per combined analysis in their comments). Two CronJobs, one per namespace, each pushing to its own Borg repo on the newly booked Storage Box (u641795.your-storagebox.de:23, BX11 1TB) with its own repo passphrase - a leaked passphrase for one doesn't expose the other: - matrix: synapse-backup dumps the synapse + matrixauthenticationservice DBs (shared postgres, existing chart-generated POSTGRES_ADMIN_PASSWORD) and the Synapse media_store PVC (read-only mount) - authentik: authentik-backup dumps the authentik DB (existing authentik-credentials pg-password) Custom image (rohana.axion1337.de/sorb/axion-backup:v1, postgres:17-alpine + borgbackup + openssh-client - matches the live Postgres major version exactly, unlike Alpine's stock postgresql16-client) pushed as a public package, same pattern as the existing threadnet-web image (no imagePullSecret needed). SSH host key pinned via known_hosts ConfigMap (captured via ssh-keyscan ahead of time) rather than trusting on first connect in an unattended job. Retention: 7 daily / 4 weekly / 6 monthly via borg prune. Schedule: 03:00 and 03:15, offset to avoid resource contention. NetworkPolicy: added each backup job's pod as an allowed source to its namespace's existing postgres ingress rule (matrix's allow-ingress-postgres, authentik's allow-ingress-authentik-postgresql from #37). Egress already unrestricted in both namespaces, so no change needed for the outbound SSH connection to the Storage Box. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-28 12:00:00 +00:00