Almost every tracked file in the repo had drifted to mode 777 on disk (only files created fresh this session were unaffected), and a chunk of that drift had already been committed as spurious +x bits on plain YAML/Markdown files (authentik.yaml, kustomization.yaml, coturn.yaml, element-server-suite.yaml, TASKS.md, install.md, etc.) - none of these need to be executable. Restored to 644 for regular files, 755 only for actual scripts (postCreateCommand.sh, docker-init.sh, install-hooks.sh, pre-commit hook, element-setup-linux.sh). Also found element-setup-macos.command was missing +x despite having a shebang and being meant for double-click execution on macOS - fixed. Added .gitignore for .DS_Store and .claude/ and stopped tracking the five .DS_Store files that had been committed by accident. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
600 lines
27 KiB
Markdown
600 lines
27 KiB
Markdown
# aXion1337.Chat – Task List & Meilensteine
|
||
|
||
**Last Updated**: 2026-07-28
|
||
**Statusübersicht**: [✅ 13 Abgeschlossen] [🔄 0 In Progress] [📋 8+ Pending] [🔒 10 Security]
|
||
|
||
---
|
||
|
||
## 📊 Status Summary (Quick View)
|
||
|
||
| Kategorie | Count | Status | Details |
|
||
|-----------|-------|--------|---------|
|
||
| **Completed** | 13 | ✅ Done | K3S, Flux, ESS, Themes, Desktop, Monitoring, TURN, Authentik (Deploy+Enrollment/Recovery/2FA), Firewall, SSH, coturn Fix, Element Call Fork, NetworkPolicies |
|
||
| **In Progress** | 0 | 🔄 — | — |
|
||
| **Backlog** | 8+ | 📋 Pending | DB Backups, PostgreSQL Migration, MAS-Template-Link, VP9-Retry |
|
||
| **Security Tasks** | 5 | 🔒 Pending | auditd, Kernel hardening, CrowdSec, Falco, WAF |
|
||
|
||
### Priority Distribution
|
||
|
||
| Priority | Count | Timeline |
|
||
|----------|-------|----------|
|
||
| 🔴 **CRITICAL** | 3 | This week |
|
||
| 🟠 **HIGH** | 4 | 1–2 weeks |
|
||
| 🟡 **MEDIUM** | 8 | ~1 month |
|
||
| 🟢 **LOW** | 4+ | Nice-to-have |
|
||
|
||
---
|
||
|
||
## 🗓️ Session-Zusammenfassung 2026-07-27/28 (fortlaufend aktualisiert)
|
||
|
||
Nach längerer Pause wiederaufgenommen — Mac war neu aufgesetzt, Zugriff (SSH, Kubeconfig,
|
||
age-Key, Homebrew/flux/helm/sops/age) komplett wiederhergestellt und dauerhaft in `~/.zshrc`
|
||
verankert. Was in dieser Session erledigt wurde:
|
||
|
||
1. **Authentik Enrollment/Recovery/2FA** (Issue #7 ✅ geschlossen) — siehe Phase 8 unten und
|
||
`docs/troubleshooting/README.md`. `matrix-invitation`- und `matrix-recovery`-Flows waren
|
||
kaputt bzw. leer, jetzt als Authentik Blueprint (`apps/authentik/authentik-blueprints.yaml`)
|
||
deklarativ repariert. E2E mit echten Test-Usern (`clark`, `lucky`) verifiziert.
|
||
2. **coturn-Crash behoben** — Liveness-Probe nutzte `netstat` (existiert nicht im Image),
|
||
Server killte einen gesunden Prozess seit 88 Tagen, 36.000+ Restarts. Auf `tcpSocket`-Probe
|
||
umgestellt, läuft seitdem stabil.
|
||
3. **Element Call Fork** (Issue #8 ✅ geschlossen, Release `m6-element-call-fork-complete`) —
|
||
1440p/60fps-Defaults, siehe Kapitel 4 in `docs/deployment-guides/04-element-customization.md`.
|
||
**Wichtig**: erzwungenes `video_codec: vp9` hat Calls kurzzeitig live komplett kaputt gemacht
|
||
(kein Bild/Ton) — sofort zurückgerollt, ohne Codec-Zwang läuft's. Root Cause dafür nicht
|
||
abschließend isoliert, nur umgangen.
|
||
4. **Identitäts-Aufräumarbeiten**: `sorB`'s Authentik-E-Mail korrigiert (`thorec@hotmail.de`),
|
||
MAS OIDC-Link (`upstream_oauth_links`) von `sorB` zeigte fest auf den alten MAS-User
|
||
`akadmin`/`@akadmin:axion1337.chat` (Sub-Hash ist stabil über Username-Renames, daher blieb
|
||
die Verknüpfung nach dem Rename "akadmin"→"sorB" bestehen) — umgehängt auf `sorb`/
|
||
`@sorb:axion1337.chat`. Neue Identität `elbojoloco` angelegt (E-Mail `cfx@riot.8shield.net`),
|
||
verknüpft mit dem alten `akadmin`-MAS-User. **Übrig**: ein leeres, unverknüpftes
|
||
`@bojeledoggo:axion1337.chat`-Konto (Tippfehler-Artefakt) — User räumt das selbst auf.
|
||
5. **NetworkPolicies** (Issue #10 ✅ geschlossen) — siehe "Network Security" Abschnitt unten.
|
||
Zwei Live-Incidents beim Rollout (Port-Verwechslungen), beide binnen Minuten live gepatcht
|
||
und danach committed. Nebenbei: `matrixRTC`-Authorisation-Service OOM-Fix (20Mi→128Mi).
|
||
6. **Element Call Qualität nachgeschärft** — 720p-Zwischen-Simulcast-Layer ergänzt (sonst
|
||
harter Sprung von 1440p auf blockiges 360p bei kleinsten Netzwerkschwankungen), und
|
||
`video_codec: h264` statt VP8 (klassisches Simulcast wie VP8, kein SVC-Risiko wie bei
|
||
VP9, oft hardwarebeschleunigt v.a. auf iOS). Live verifiziert: 7/8 Tracks nativ H.264,
|
||
1 sauberer VP8-Fallback. Deployed als `v0.2.3-elementcall-h264`.
|
||
7. **Backlog nach Gitea migriert** — restlicher offener Backlog (VP9-Retry, ThreadNet-Web-Bug,
|
||
MAS-Template-Link, WAF und 17 weitere Security-/Infra-Punkte) als Issues #11–#31 angelegt,
|
||
veraltete erledigte Punkte (Authentik Stage 2/E2E-Test/Invite-Links, Hetzner-Firewall,
|
||
SSH-Hardening) aus dieser Datei entfernt bzw. als done markiert.
|
||
|
||
### Offene Punkte
|
||
- **VP9-Retry**: vermutete Ursache jetzt bekannt (LiveKit nutzt SVC für vp9/av1, Fork-Code
|
||
setzt aber immer Simulcast-Layer) — braucht einen Code-Fix in `buildPublishOptions()`
|
||
(`src/livekit/options.ts`) bevor erneut versucht wird. Stattdessen H.264 probiert (siehe
|
||
unten) — läuft gut, kein SVC-Risiko, hardwarebeschleunigt auf mehr Geräten.
|
||
- **`ThreadNet-Web` Build-Bug**: `scripts/docker-link-repos.sh`/`docker-package.sh` nicht
|
||
ausführbar committet + veralteter `matrix-js-sdk#develop`-Pin im Lockfile blockiert
|
||
vollständigen Neu-Build des Web-Forks. Noch nicht gefixt, User hat noch nicht final
|
||
entschieden ob gewünscht.
|
||
- **Verwaistes `@bojeledoggo:axion1337.chat`**: leeres Matrix-Konto ohne OIDC-Link, User räumt
|
||
das selbst auf (braucht dafür seinen eigenen Access-Token für die Admin-API).
|
||
- **MAS-Template-Link**: 2FA/Passkey-Setup-Links direkt auf `account.axion1337.chat/account/`
|
||
statt nur über `docs/setup/security.html` — braucht MAS Custom-Template-Override
|
||
(`templates.path`), größerer separater Task.
|
||
- Nächste Kandidaten aus den offenen Issues: #6 (DB-Backup, CRITICAL), #9 (PostgreSQL-Migration),
|
||
#10 (NetworkPolicies).
|
||
|
||
---
|
||
|
||
## 🎯 Next Steps (Priorisiert)
|
||
|
||
### 🔴 **THIS WEEK – CRITICAL**
|
||
1. ✅ **Authentik Stage 2 abschließen**
|
||
- ✅ Manual: OIDC Provider + Application in Authentik UI erstellt
|
||
- ✅ Code: `upstream_oauth2_config` in `mas-secret.yaml` eingefügt
|
||
- ✅ Code: `passwords: enabled: false` aktiviert
|
||
- ✅ Commit: `cdfbf7d` - Enable Authentik OIDC integration in MAS
|
||
- ✅ Verified: Login mit Authentik funktioniert (2026-05-15)
|
||
- **Status**: COMPLETE
|
||
|
||
2. ✅ **Hetzner Cloud Firewall – Optimierte Konfiguration**
|
||
- ✅ Ingress: 80/443 + TURN/STUN + RTC Services
|
||
- ✅ SSH: Spezifische IPs (port 2248, nicht 22)
|
||
- ✅ Default-Deny für nicht-definierte Traffic
|
||
- **Status**: COMPLETE (optimiert über Plan)
|
||
|
||
3. ✅ **SSH Hardening**
|
||
- ✅ PasswordAuthentication: no (key-only)
|
||
- ✅ PermitRootLogin: no (root disabled)
|
||
- ✅ MaxAuthTries: 3 (verified 2026-05-15)
|
||
- **Status**: COMPLETE
|
||
|
||
4. **Database Backup Strategy – Decision & First Backup**
|
||
- ⏳ Decision: CloudNativePG (on K3S) or Hetzner Postgres (managed)?
|
||
- ⏳ Setup: Daily automated backups
|
||
- ⏳ Setup: Off-site storage (S3 / Storage Box)
|
||
- ⏳ Setup: Monthly verified restores
|
||
- Est. Time: 2–3 days
|
||
- Priority: CRITICAL (disaster recovery)
|
||
- **Status**: NEXT
|
||
|
||
### 🟠 **NEXT 1–2 WEEKS – HIGH**
|
||
1. **Authentik End-to-End Test**
|
||
- Test: Login flow Element → MAS → Authentik → Matrix User
|
||
- Test: Password reset
|
||
- Create: Test invite links
|
||
- Est. Time: 2 hours
|
||
|
||
2. **Element Call Fork**
|
||
- Fork: element-hq/element-call
|
||
- Feature: Video/audio constraints parameters
|
||
- Integration: Synapse well-known config
|
||
- Est. Time: 2–3 days
|
||
|
||
3. **External PostgreSQL Migration**
|
||
- Decision: CloudNativePG vs. Hetzner Postgres
|
||
- Setup: HA + Replication
|
||
- Migration: Move data from ESS embedded Postgres
|
||
- Testing: Verify all services work
|
||
- Est. Time: 1–2 days
|
||
|
||
4. **NetworkPolicies Deployment**
|
||
- Create: Default-Deny for `matrix` namespace
|
||
- Create: Allow rules (Synapse↔Postgres, MAS↔Postgres, Ingress→Web, etc.)
|
||
- Test: Ensure no service breakage
|
||
- Est. Time: 1 day
|
||
|
||
---
|
||
|
||
## ✅ Abgeschlossene Aufgaben (Chronologisch)
|
||
|
||
### Phase 1: Basis-Setup
|
||
- [x] **K3S Cluster aufsetzen** – Single-Node auf Hetzner Cloud (49.13.132.245)
|
||
- Commit: `initial-setup` (vor Projekt)
|
||
- Status: ✅ Läuft
|
||
|
||
- [x] **Flux CD Installation**
|
||
- SOPS + age Encryption
|
||
- GitOps Repository konfigurieren
|
||
- Commit: `setup-flux` (vor Projekt)
|
||
- Status: ✅ Läuft
|
||
|
||
- [x] **Element Server Suite v26.4.0 Deployment**
|
||
- Synapse Homeserver (`matrix.axion1337.chat`)
|
||
- Matrix Authentication Service (`account.axion1337.chat`)
|
||
- Element Web (`axion1337.chat`)
|
||
- Element Admin (`admin.axion1337.chat`)
|
||
- MatrixRTC/Element Call (`mrtc.axion1337.chat`)
|
||
- Commit: `deploy-ess-matrix-stack`
|
||
- Status: ✅ Running
|
||
|
||
### Phase 2: Core Features
|
||
- [x] **7 Custom Element Web Themes**
|
||
- aXion1337 Dark, Deep Purple, Discord Dark, Electric Blue, Everforest, Gruvbox, Wal
|
||
- Alphabetisch sortiert
|
||
- Commit: `add-custom-element-themes`
|
||
- Status: ✅ Deployed
|
||
|
||
- [x] **Element Desktop Setup Scripts** (Windows/macOS/Linux)
|
||
- Auto-Download + Install + Config
|
||
- Hosted auf `axion1337.chat/docs/setup/`
|
||
- Commits: `add-element-desktop-setup-scripts`, `fix-element-setup-script-hosting`
|
||
- Status: ✅ Deployed
|
||
|
||
- [x] **Room Policies**
|
||
- Message Retention (1d–1y lifecycle)
|
||
- Room Publication Rules (allow all)
|
||
- Auto-Join Rooms für Onboarding
|
||
- Commit: `add-synapse-retention-publication-autojoin`
|
||
- Status: ✅ Deployed
|
||
|
||
### Phase 3: WebRTC & Medienübertragung
|
||
- [x] **TURN Server (coturn) für Video-Calls**
|
||
- Domain: `turn.axion1337.chat`
|
||
- HMAC-Auth mit Shared Secret
|
||
- Ports: 3478/udp, 3478/tcp, 5349/tcp, 49152-65535/udp
|
||
- Commit: `implement-turn-server-coturn-for-webrtc-video-calls`
|
||
- Status: ✅ Deployed
|
||
- Manual: DNS A-Record + Firewall-Ports öffnen (noch erforderlich)
|
||
|
||
### Phase 4: Monitoring & Observability
|
||
- [x] **Monitoring Stack Integration**
|
||
- Alloy (Grafana Agent) als Collector
|
||
- Remote Write zu Selendis (10.0.0.3:9090 Prometheus, :3100 Loki)
|
||
- kube-state-metrics, node-exporter DaemonSet
|
||
- Commits: `integrate-monitoring-alloy-prometheus-loki`, `fix-prometheus-remote-write-docker`
|
||
- Status: ✅ Deployed
|
||
|
||
### Phase 5: Identity Provider (Authentik)
|
||
- [x] **Authentik Stage 1 Deployment**
|
||
- HelmRelease v2026.x in `authentik` namespace
|
||
- Embedded PostgreSQL + Alloy-compatible
|
||
- Cert-Manager für TLS
|
||
- Commit: `deploy-authentik-as-identity-provider-for-matrix-stage-1`
|
||
- Status: ✅ Deployed
|
||
- Manual: Admin-Passwort + OIDC Provider + Application + Enrollment Flow erstellt ✅
|
||
|
||
- [x] **Authentik Stage 2 – MAS Integration**
|
||
- ✅ Authentik Admin UI: OIDC Provider erstellt (Authentik)
|
||
- ✅ Authentik Admin UI: Application mit Slug `matrix` erstellt
|
||
- ✅ Authentik Admin UI: Enrollment Flow mit Invitation Stage konfiguriert
|
||
- ✅ Client ID + Secret kopiert
|
||
- ✅ MAS `upstream_oauth2_config` mit Client Credentials aktualisiert
|
||
- ✅ `passwords: enabled: false` aktiviert
|
||
- ✅ Commit: `cdfbf7d` - Enable Authentik OIDC integration in MAS
|
||
- ✅ Verified: Login mit Authentik funktioniert (2026-05-15)
|
||
- Status: ✅ Deployed & Verified
|
||
|
||
### Phase 6: Dokumentation
|
||
- [x] **Deployment Guides erstellen**
|
||
- 5 Markdown-Dateien in `docs/deployment-guides/`
|
||
- Chronologisch geordnet
|
||
- Troubleshooting + Best Practices
|
||
- Commit: `add-comprehensive-deployment-configuration-documentation`
|
||
- Status: ✅ Deployed
|
||
|
||
- [x] **Gitea Wiki erstellen**
|
||
- Home.md mit Navigation
|
||
- Alle Deployment Guides in Root
|
||
- Operations + Archive Dokumentation
|
||
- Wiki Branch gepusht zu rohana.axion1337.de
|
||
- Status: ✅ Live
|
||
|
||
- [x] **Gitea Issues & Project Board**
|
||
- 8 Issues erstellt (#3-#10): 4 CRITICAL + 4 HIGH
|
||
- Priority Labels: critical, high
|
||
- Area Labels: authentik, security, database, infrastructure, element
|
||
- Status: ✅ Tracking
|
||
|
||
### Phase 7: Infrastructure Security (Critical)
|
||
- [x] **Hetzner Cloud Firewall Configuration**
|
||
- SSH: Spezifische IPs (port 2248)
|
||
- HTTP/HTTPS: Any IPv4/IPv6
|
||
- TURN/STUN: WebRTC Ports
|
||
- RTC Services: SFU + Auth Ports
|
||
- Status: ✅ Optimiert & Deployed
|
||
|
||
- [x] **SSH Hardening**
|
||
- PasswordAuthentication: no (key-only)
|
||
- PermitRootLogin: no
|
||
- MaxAuthTries: 3
|
||
- Verified: 2026-05-15
|
||
- Status: ✅ Complete
|
||
|
||
---
|
||
|
||
## 🔄 In Progress / Blocked
|
||
|
||
**None** – Alle CRITICAL Tasks erledigt! Nächster Focus: Database Backups
|
||
|
||
### Phase 8: Authentik Enrollment/Recovery/MFA Fix (2026-07-27)
|
||
- [x] **matrix-invitation Flow repariert** – fehlende Write/Password/Login-Stages ergänzt, Reihenfolge korrigiert, als Authentik Blueprint (`apps/authentik/authentik-blueprints.yaml`) reproduzierbar gemacht
|
||
- [x] **matrix-invitation-prompt** – 16 fehlerhafte `validation_policies` entfernt (crashten mit `AnonymousUser`/`NoneType`-Fehlern)
|
||
- [x] **Redirect-Stage** – Flow endet jetzt auf `axion1337.chat` statt in der `/if/user/`-Sackgasse (blockiert für `type=external`)
|
||
- [x] **matrix-recovery Flow gebaut** – war komplett leer (0 Stages); Passwort-Reset funktioniert jetzt, verlinkt von der echten Login-Seite
|
||
- [x] **Brand.default_application gesetzt** – behebt mehrere Dead-Ends, wenn eingeloggte User `/` ohne Ziel aufrufen
|
||
- [x] **2FA/Passkey Selbst-Einrichtung** – Links zu `default-authenticator-totp-setup`/`-webauthn-setup` (2FA bleibt optional, `not_configured_action=skip`), dokumentiert unter `axion1337.chat/docs/setup/security.html`
|
||
- [ ] **Backlog**: → **Issue #13** (MAS Custom-Template-Override für 2FA/Passkey-Link auf `account.axion1337.chat/account/`)
|
||
|
||
---
|
||
|
||
## 📋 Backlog (Weitere Aufgaben)
|
||
|
||
**Ab 2026-07-28 in Gitea-Issues gepflegt statt hier** (eine Quelle der Wahrheit) — offene Issues:
|
||
[#6](https://rohana.axion1337.de/sorb/axion1337.chat-gitops/issues/6) DB-Backup-Strategie,
|
||
[#9](https://rohana.axion1337.de/sorb/axion1337.chat-gitops/issues/9) Externe PostgreSQL-Migration,
|
||
[#11](https://rohana.axion1337.de/sorb/axion1337.chat-gitops/issues/11)–[#31](https://rohana.axion1337.de/sorb/axion1337.chat-gitops/issues/31)
|
||
(VP9-Retry, ThreadNet-Web-Build-Bug, MAS-Template-Link, WAF, Media-PVC-Backups, Pod Security
|
||
Admission, Federation-Allowlist, Mjolnir/Draupnir, Content-Scanner, External-Secrets,
|
||
Renovate/Trivy, Security-Advisory-Monitoring, automountServiceAccountToken,
|
||
unattended-upgrades, K3s-API-Security, auditd, Kernel-Hardening, Lynis, CrowdSec, Falco).
|
||
Die detaillierten Beschreibungen unten sind das historische Original, aus dem die Issues
|
||
entstanden sind — nicht mehr getrennt pflegen, stattdessen die Issues aktuell halten.
|
||
|
||
### Element Call Enhancement
|
||
- [x] **Element Call Fork für Custom Constraints** (2026-07-28, Closes #8)
|
||
- Fork: `rohana.axion1337.de/sorb/threadnet-call` (basiert auf `emmick4/element-call:livekit`,
|
||
das den noch nicht gemergten Upstream-PR element-hq/element-call#3736 enthält —
|
||
config-driven `media_quality`, keine Custom-Logik nötig)
|
||
- Defaults angehoben: Video bis 1440p/60fps (~8 Mbps), Screen-Share 1440p/30fps (~6 Mbps).
|
||
Das sind Startwerte, keine harten Limits — Nutzer können in den Settings weiter hochdrehen.
|
||
- **Incident (2026-07-28)**: Erster Deploy (`v0.2.0`, mit `video_codec: vp9` erzwungen) hat
|
||
Calls komplett kaputt gemacht (kein Bild/Ton), obwohl LiveKit-Server-Logs den
|
||
Codec-Regression-Fallback auf VP8 als erfolgreich zeigten — Root Cause nicht abschließend
|
||
isoliert. Sofort auf `v0.1.0` zurückgerollt, dann `v0.2.1` ohne erzwungenen Codec (Standard
|
||
VP8) mit denselben 1440p/60fps-Werten deployed und vom Nutzer live bestätigt: funktioniert.
|
||
VP9-Präferenz vorerst fallengelassen, siehe Backlog.
|
||
- Rauschunterdrückung: nur clientseitige WebRTC-Standardtoggles (echoCancellation/
|
||
noiseSuppression/autoGainControl), kommt kostenlos mit derselben PR. **Bewusst kein**
|
||
server-seitiges ML-Noise-Cancellation (LiveKit Agents + DTLN/RNNoise) — laut LiveKits
|
||
eigener Doku ist das für Mensch-zu-Mensch-Calls der falsche Ansatz (nur für AI-Voice-Agents
|
||
gedacht, kein Standard-Pfad um bereinigtes Audio an andere Teilnehmer zurückzugeben).
|
||
- Well-Known/`org.matrix.msc4143.rtc_foci`-Delegation war schon vom ESS-Chart korrekt
|
||
automatisch konfiguriert — kein Handlungsbedarf trotz anderslautendem Issue-Text.
|
||
- **Deployment-Ansatz geändert**: `sorb/ThreadNet-Web` (der Element-Web-Fork) hat einen
|
||
vorbestehenden, unabhängigen Build-Bug (siehe unten) und ließ sich nicht komplett neu
|
||
bauen. Stattdessen: nur der `/app/widgets/element-call/`-Ordner im bereits laufenden
|
||
`threadnet-web:v0.1.0`-Image ausgetauscht → neues Image
|
||
`rohana.axion1337.de/sorb/threadnet-web:v0.2.0-elementcall-mediaquality`.
|
||
- Verifiziert: `media_quality` live auf `axion1337.chat/widgets/element-call/config.json`.
|
||
- **Gefunden, nicht gefixt**: `ThreadNet-Web` lässt sich aktuell nicht komplett neu bauen
|
||
— `scripts/docker-link-repos.sh`/`docker-package.sh` sind im Repo nicht ausführbar
|
||
committet (Mode 644 statt 755), UND der gepinnte `matrix-js-sdk#develop`-Commit im
|
||
Lockfile ist zu alt (fehlt `src/oidc/authorize.ts`, das `apps/web` importiert). Beides
|
||
unabhängig von diesem Fix, blockiert aber jeden zukünftigen vollständigen Rebuild.
|
||
- Backlog: MAL-basierte Noise-Cancellation (LiveKit Agents + self-hosted DTLN/RNNoise) als
|
||
experimentelle Idee, falls später gewünscht — kein etablierter Pfad für Conferencing.
|
||
- Backlog: VP9-Codec-Präferenz erneut versuchen, sobald PR #3736 upstream gemerged/gereift
|
||
ist oder Root Cause des Ausfalls isoliert wurde (Browser-Konsolen-Repro nötig).
|
||
|
||
### Database Hardening
|
||
- [ ] **External/Dedicated PostgreSQL Deployment**
|
||
- Option 1: CloudNativePG Operator (open-source, auf K3S)
|
||
- Option 2: Managed Hetzner Postgres
|
||
- Separate aus ESS matrix-stack embedded Postgres
|
||
- HA + Replication
|
||
- Est. Effort: 1–2 days
|
||
- Priority: **HIGH** (reliability)
|
||
|
||
- [ ] **Database Backup Strategy**
|
||
- Daily automated backups (PgBackRest oder velero)
|
||
- Off-site backup storage (S3 / Hetzner Storage Box)
|
||
- Monthly verified restores (test restore → verify data integrity)
|
||
- Backup + restore documentation
|
||
- Est. Effort: 2–3 days
|
||
- Priority: **CRITICAL** (disaster recovery)
|
||
|
||
- [ ] **Synapse Media PVC Backups**
|
||
- Separate backup pipeline für `/data/media_store` PVC
|
||
- Reason: Media oft >100GB, sollte nicht im DB-Backup sein
|
||
- Velero + Restic für block-level backup
|
||
- Est. Effort: 1 day
|
||
- Priority: **HIGH** (data preservation)
|
||
|
||
### Network Security
|
||
- [x] **NetworkPolicies – K8s-Layer Segmentation** (2026-07-28, Closes #10)
|
||
- Default-Deny Ingress (egress left untouched) für `matrix` UND `authentik` namespaces,
|
||
per-Komponente Allow-Regeln in `apps/authentik/networkpolicy.yaml` und
|
||
`apps/production/networkpolicy.yaml`. Rollout: authentik zuerst als Pilot, dann matrix.
|
||
- Empirisch verifiziert, dass K3s' eingebauter NetworkPolicy-Controller tatsächlich
|
||
durchsetzt (Testnamespace, Timeout- statt Refused-Verhalten unter Deny-Policy).
|
||
- **Zwei Live-Incidents beim Rollout, beide binnen Minuten behoben**:
|
||
1. `authentik-server`: Regel erlaubte Service-Port 80/443, aber NetworkPolicy filtert
|
||
auf dem tatsächlichen Container-Port (9000/9443 nach kube-proxy-DNAT) — 502 auf
|
||
`auth.axion1337.chat`, sofort korrigiert.
|
||
2. `matrix-authentication-service`: Regel erlaubte Synapse nur auf Port 8081, aber
|
||
Synapse ruft `/oauth2/introspect` tatsächlich auf **Port 8080** — jede
|
||
authentifizierte Anfrage (inkl. `/sync`) scheiterte mit 503, alle Clients zeigten
|
||
"Verbindung unterbrochen". Live gepatcht, dann committed.
|
||
- Lehre für zukünftige NetworkPolicies in diesem Repo: wo immer ein Service benannte
|
||
Ports (`targetPort: <name>`) nutzt, diese direkt in der Policy referenzieren statt
|
||
Portnummern zu raten — schließt genau diese Fehlerklasse aus.
|
||
- Nebenbefund (unabhängig von NetworkPolicies): `matrixRTC`-Authorisation-Service hatte
|
||
ein 20Mi-Memory-Limit (Chart-Default), OOM-gekillt nach ~74 Tagen Uptime während der
|
||
Verifikations-Calls — auf 64Mi/128Mi angehoben.
|
||
- `coturn` (hostNetwork) bewusst ausgenommen — NetworkPolicy greift dort nicht.
|
||
- `authentik-postgresql`'s Bitnami-Chart-Policy (Port 5432, quelloffen) bewusst nicht
|
||
angefasst/dupliziert, da Helm-verwaltet.
|
||
|
||
- [ ] **Pod Security Admission (Restricted)**
|
||
- Apply to `matrix` & `authentik` namespaces
|
||
- Enforce: non-root, no privileged, read-only root fs
|
||
- Test: Ensure no chart breakage
|
||
- Est. Effort: 1 day
|
||
- Priority: **MEDIUM** (hardening)
|
||
|
||
### Federation & Access Control
|
||
- [ ] **Federation-Allowlist oder Closed Federation**
|
||
- Decision: Which servers to federate with?
|
||
- If allowlist: explicit `federation_domain_whitelist`
|
||
- If closed: `allow_public_rooms_without_join_rules: false`
|
||
- Synapse config in `synapse-values.yaml`
|
||
- Est. Effort: 4 hours
|
||
- Priority: **MEDIUM** (security policy)
|
||
|
||
### Moderation & Anti-Abuse
|
||
- [ ] **Mjolnir/Draupnir Bot Deployment**
|
||
- Open-source moderation bot für Matrix
|
||
- Reason: Invitation-based, aber Federation kann Spam bringen
|
||
- Auto-ban known bad servers/users
|
||
- Spam-detection rules
|
||
- HelmChart oder custom Deployment
|
||
- Est. Effort: 1–2 days
|
||
- Priority: **MEDIUM** (ops safety)
|
||
|
||
- [ ] **Content Scanner for Media**
|
||
- matrix-content-scanner + ClamAV antivirus
|
||
- Scan uploaded media for malware
|
||
- Block suspicious files
|
||
- Est. Effort: 1–2 days
|
||
- Priority: **LOW–MEDIUM** (optional but good practice)
|
||
|
||
### Secrets Management
|
||
- [ ] **External-Secrets Operator oder SOPS für Flux**
|
||
- Current: SOPS with age encryption
|
||
- Consideration: External-Secrets for cloud-native (AWS Secrets Manager, Hetzner Vault, etc.)
|
||
- OR: Improve SOPS rotation strategy
|
||
- Decision needed: Keep SOPS or upgrade?
|
||
- Est. Effort: 2–3 days (if switching)
|
||
- Priority: **LOW** (current SOPS setup working)
|
||
|
||
### Image & Dependency Management
|
||
- [ ] **Renovate / Dependabot Setup**
|
||
- Auto-update Helm Chart versions
|
||
- Auto-update Container Image Tags
|
||
- Monitor for security patches
|
||
- Est. Effort: 4 hours
|
||
- Priority: **MEDIUM** (maintenance)
|
||
|
||
- [ ] **Trivy Image Scanning**
|
||
- Scan images in Flux HelmReleases for CVEs
|
||
- Block deployment if critical CVE found
|
||
- CI/CD hook in git workflow
|
||
- Est. Effort: 8 hours
|
||
- Priority: **LOW–MEDIUM** (security posture)
|
||
|
||
- [ ] **Monitor ESS & Element Security Advisories**
|
||
- Subscribe to `element-hq` security mailing list
|
||
- Monitor `#matrix-community` security channels
|
||
- Auto-alerts on new CVEs/patches
|
||
- Est. Effort: Ongoing (low maintenance)
|
||
- Priority: **MEDIUM** (security awareness)
|
||
|
||
### Container Security
|
||
- [ ] **Disable automountServiceAccountToken Everywhere**
|
||
- Audit all Deployments/StatefulSets
|
||
- Disable for: Synapse, ElementWeb, MAS, Postgres, Authentik (where not needed)
|
||
- Add `automountServiceAccountToken: false` to spec.template.spec
|
||
- Test: Ensure no breakage
|
||
- Est. Effort: 4 hours
|
||
- Priority: **MEDIUM** (least-privilege)
|
||
|
||
---
|
||
|
||
## 🔒 Security Hardening (Host & Cluster Level)
|
||
|
||
### Host OS Layer (Ubuntu/Debian)
|
||
- [x] **Hetzner Cloud Firewall** – Default-Deny inbound, siehe "Phase 7" oben. **Done.**
|
||
- [x] **SSH Hardening** – Key-only, Root-Login disabled, Port 2248, siehe "Phase 7" oben. **Done.**
|
||
|
||
- [ ] **unattended-upgrades**
|
||
- Enable automatic security updates
|
||
- Configure: APT::Periodic::Update-Package-Lists "1";
|
||
- Configure: APT::Periodic::Unattended-Upgrade "1";
|
||
- Configure: APT::Periodic::AutocleanInterval "7";
|
||
- Est. Effort: 30 min
|
||
- Priority: **HIGH** (set & forget)
|
||
|
||
- [ ] **K3S API Security**
|
||
- Current: K3S API listening on :6443 on all interfaces (default)
|
||
- Hardening:
|
||
- Option 1: Firewall restrict :6443 to localhost only
|
||
- Option 2: K3S --bind-address + --advertise-address to WireGuard IP
|
||
- Option 3: kubectl access only via jumphost/bastion
|
||
- Est. Effort: 2 hours
|
||
- Priority: **HIGH** (API is high-value target)
|
||
|
||
- [ ] **auditd for File Integrity & Syscall Audit**
|
||
- Monitor: /etc, ~/.kube, /var/lib/rancher/k3s
|
||
- Audit rules für sensitive file changes
|
||
- Low overhead, good signal/noise ratio
|
||
- Output to syslog / centralized logging
|
||
- Est. Effort: 2 hours
|
||
- Priority: **MEDIUM** (forensics + compliance)
|
||
|
||
- [ ] **Kernel Hardening (sysctl)**
|
||
- Apply hardening recommendations from Lynis
|
||
- Key settings:
|
||
- kernel.kptr_restrict=2 (hide kernel pointers)
|
||
- kernel.dmesg_restrict=1 (restrict dmesg)
|
||
- net.ipv4.tcp_syncookies=1 (SYN flood protection)
|
||
- net.ipv4.conf.all.rp_filter=1 (reverse path filtering)
|
||
- net.ipv4.conf.all.send_redirects=0
|
||
- net.ipv6.conf.all.disable_ipv6=0 (or =1 if no IPv6 needed)
|
||
- Persist via /etc/sysctl.d/99-hardening.conf
|
||
- Est. Effort: 2 hours
|
||
- Priority: **MEDIUM** (defense in depth)
|
||
|
||
- [ ] **Lynis Security Baseline**
|
||
- Run `lynis audit system`
|
||
- Review recommendations
|
||
- Implement high-priority findings
|
||
- Aim for score >80
|
||
- Re-run quarterly
|
||
- Est. Effort: 4 hours (initial) + 1 hour quarterly
|
||
- Priority: **MEDIUM** (baseline verification)
|
||
|
||
### Cluster Layer (K3S / Kubernetes)
|
||
- [ ] **CrowdSec Integration**
|
||
- Install CrowdSec agent on host
|
||
- Connect to CrowdSec Hub (commercial platform, free tier available)
|
||
- Feed auth.log, syslog → CrowdSec for attack detection
|
||
- Auto-block IPs via local firewall or Hetzner Firewall API
|
||
- Est. Effort: 4 hours
|
||
- Priority: **MEDIUM** (proactive threat response)
|
||
|
||
- [ ] **Falco Runtime Monitoring**
|
||
- Install Falco DaemonSet in K3S
|
||
- Monitor: Shell spawning in containers, suspicious syscalls, privilege escalation
|
||
- Output to Loki / syslog
|
||
- Alert on anomalies
|
||
- Est. Effort: 1 day
|
||
- Priority: **MEDIUM** (runtime detection)
|
||
|
||
---
|
||
|
||
## 🎯 Meilensteine (Milestones)
|
||
|
||
| Meilenstein | Beschreibung | Status | ETA |
|
||
|------------|-------------|--------|-----|
|
||
| **M1: Basis-Setup** | K3S + Flux + ESS deployed | ✅ Done | - |
|
||
| **M2: Core Matrix** | Themes, Scripts, Policies | ✅ Done | - |
|
||
| **M3: WebRTC & Monitoring** | TURN + Alloy/Prometheus/Loki | ✅ Done | - |
|
||
| **M4: Identity Provider** | Authentik Stage 1+2 (pending Stage 2) | 🔄 In Progress | ~1–2 days |
|
||
| **M5: Production-Ready** | DB Backups, NetworkPolicies, Security Hardening | 📋 Backlog | ~2–3 weeks |
|
||
| **M6: Advanced Features** | Element Call Fork, Content Scanner, Mjolnir | 📋 Backlog | ~4+ weeks |
|
||
| **M7: Enterprise-Ready** | Full compliance (DSGVO), HA setup, Disaster Recovery | 🎯 Future | ~8+ weeks |
|
||
|
||
---
|
||
|
||
## 📊 Prioritäts-Kategorien
|
||
|
||
### 🔴 CRITICAL (do immediately)
|
||
- Hetzner Cloud Firewall setup
|
||
- Database backup strategy
|
||
- SSH hardening
|
||
|
||
### 🟠 HIGH (do within 1–2 weeks)
|
||
- Authentik Stage 2 completion
|
||
- External PostgreSQL migration
|
||
- NetworkPolicies
|
||
- Element Call fork
|
||
|
||
### 🟡 MEDIUM (do within 1 month)
|
||
- CrowdSec + Falco
|
||
- Mjolnir bot
|
||
- Renovate/Trivy
|
||
- PSA restricted mode
|
||
- Kernel hardening
|
||
|
||
### 🟢 LOW (nice-to-have, do if time allows)
|
||
- Content scanner (ClamAV)
|
||
- External-Secrets upgrade
|
||
- SSH port relocation
|
||
- Advanced federation rules
|
||
|
||
---
|
||
|
||
## 📝 Notes & Decision Points
|
||
|
||
### Authentik Stage 2 Blocker
|
||
⏳ **Waiting for**: User to manually configure Authentik OIDC Provider in Authentik Admin UI.
|
||
- Once done, provide Client ID + Secret
|
||
- Then: Commit Stage 2 MAS config
|
||
|
||
### Database: CloudNativePG vs. Hetzner Postgres
|
||
- **CloudNativePG**: Open-source, runs on K3S, full control
|
||
- **Hetzner Postgres**: Managed, backups included, less ops overhead
|
||
- **Decision**: Recommend CloudNativePG for now (cost-effective), migrate to Hetzner later if operational overhead too high
|
||
|
||
### Federation: Allowlist vs. Closed?
|
||
- **Allowlist**: Default federation with all public servers, can be attacked
|
||
- **Closed**: Only federate with trusted servers (higher security, lower interop)
|
||
- **Decision**: Depends on user intent. For now: allow all, add Mjolnir for abuse protection
|
||
|
||
### Security Framework
|
||
- **Layers**: Perimeter (Firewall) → Host (SSH, auditd, hardening) → Cluster (NetworkPolicies, PSA, Falco) → App (Rate-limits, Mjolnir)
|
||
- **Approach**: Implement incrementally, test after each layer
|
||
|
||
---
|
||
|
||
## 🔗 Related Documentation
|
||
|
||
- `docs/deployment-guides/README.md` – Overview
|
||
- `docs/deployment-guides/01-turn-server-setup.md` – TURN
|
||
- `docs/deployment-guides/02-authentik-identity-provider.md` – Authentik (Stage 1 + Stage 2 plan)
|
||
- `docs/deployment-guides/03-monitoring-integration.md` – Monitoring
|
||
- `docs/deployment-guides/04-element-customization.md` – Themes, Desktop
|
||
- `docs/deployment-guides/05-room-policies.md` – Policies
|
||
|
||
---
|
||
|
||
**Last Updated**: 2026-05-14
|
||
**Next Review**: 2026-05-21
|