docs: fix stale status claims, move CLAUDE.md into repo, add host-config note
Documentation audit across the repo found several places still describing finished work as pending or in-progress (Authentik Stage 2, Element Call fork, NetworkPolicies in TASKS.md's own Next Steps section, the Boje troubleshooting entry). Also moves CLAUDE.md from the untracked parent directory into the repo root and brings its content up to date, and documents the new host-config/ pattern in README.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Thore Cimbal
co-authored by
Claude Sonnet 5
Claude Sonnet 5
parent
376772078a
commit
932443df63
@@ -0,0 +1,331 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Overview
|
||||
|
||||
This is a **GitOps-based Kubernetes deployment** of **Element Server Suite (ESS Community v26.4.0)**, a complete Matrix homeserver stack. The repository contains Infrastructure-as-Code using **FluxCD** for GitOps synchronization, with encryption (SOPS/age), service mesh (Traefik), certificate management (Cert-Manager), and auxiliary services like Authentik, TURN/coturn, Draupnir (moderation), ClamAV (content scanning), and Grafana monitoring.
|
||||
|
||||
**Key Stack Components:**
|
||||
- **K3s**: Lightweight Kubernetes distribution running on Hetzner Cloud
|
||||
- **FluxCD**: GitOps controller that watches this repository and auto-syncs changes
|
||||
- **ESS (Matrix Umbrella Chart v26.4.0)**: Synapse, MAS (Matrix Authentication Service), ElementWeb, MatrixRTC
|
||||
- **Authentik**: OIDC-based identity provider for centralized authentication, deployed both via HelmRelease and declarative Blueprints (`apps/authentik/authentik-blueprints.yaml`) for flows/OIDC-provider config that would otherwise only exist as manual admin-UI clicks
|
||||
- **Traefik**: Ingress controller (built into K3s) for routing HTTP/HTTPS traffic
|
||||
- **Cert-Manager**: Automatic TLS certificate provisioning from Let's Encrypt
|
||||
- **SOPS + age**: Transparent encryption/decryption of secrets in Git
|
||||
- **Monitoring**: Grafana Alloy (agent), Prometheus (metrics), Loki (logs)
|
||||
- **coturn**: TURN/STUN server for WebRTC audio/video calls, with monthly automated shared-secret rotation via CronJob + PR workflow
|
||||
- **Draupnir**: Matrix moderation bot (community successor to Mjolnir), ban lists/policy rooms
|
||||
- **ClamAV**: Content scanning — a Synapse module for unencrypted-room uploads, plus a standalone `clamav-http-scanner` service that a patched Element Web client (ThreadNet-Web) calls both on send and on receive, extending coverage to encrypted rooms/DMs
|
||||
- **NetworkPolicies**: default-deny-with-explicit-allow across `matrix` and `authentik` namespaces
|
||||
- **`host-config/`**: the one part of this repo that is deliberately **not** managed by Flux/GitOps — see "Host-Level (non-GitOps) Changes" below
|
||||
|
||||
## Repository Structure
|
||||
|
||||
```
|
||||
gitops/
|
||||
├── clusters/matrix/ # Flux GitRepository definition; entry point for reconciliation
|
||||
├── apps/
|
||||
│ ├── base/
|
||||
│ │ ├── infra/ # Core infrastructure (Cert-Manager, Namespaces, etc.)
|
||||
│ │ └── matrix/ # HelmRepository definition for ESS OCI chart
|
||||
│ ├── production/ # Main ESS deployment
|
||||
│ │ ├── element-server-suite.yaml # HelmRelease (ESS chart v26.4.0)
|
||||
│ │ ├── custom-configs/ # Overrides & custom configurations
|
||||
│ │ │ ├── synapse-values.yaml # Synapse customizations (ConfigMap)
|
||||
│ │ │ ├── element-values.yaml # ElementWeb customizations (ConfigMap)
|
||||
│ │ │ └── mas-secret.yaml # MAS secrets (encrypted with SOPS)
|
||||
│ │ ├── cert-issuer.yaml # Let's Encrypt ClusterIssuer
|
||||
│ │ ├── apex-ingress.yaml # Apex-domain IngressRoutes (Element Web, /_scan, etc.)
|
||||
│ │ ├── matrix-postgres-auth.yaml # PostgreSQL credentials
|
||||
│ │ ├── coturn.yaml / coturn-secret.yaml / synapse-turn-secret.yaml
|
||||
│ │ ├── turn-secret-rotation.yaml # Monthly CronJob, rotates coturn shared secret via PR
|
||||
│ │ ├── draupnir.yaml / draupnir-pvc.yaml / draupnir-secret.yaml
|
||||
│ │ ├── clamav.yaml / clamav-pvc.yaml / clamav_spam_checker.py # Synapse-side scan module
|
||||
│ │ ├── clamav-http-scanner.py / -Dockerfile / .yaml # Client-side scan service
|
||||
│ │ ├── synapse-backup.yaml / synapse-backup-secret.yaml
|
||||
│ │ └── networkpolicy.yaml # Default-deny + explicit allow rules
|
||||
│ ├── authentik/ # Identity Provider (separate namespace)
|
||||
│ │ ├── authentik.yaml # HelmRelease
|
||||
│ │ ├── authentik-blueprints.yaml # Flows/OIDC-provider as declarative code
|
||||
│ │ ├── helm-repo.yaml # HelmRepository source
|
||||
│ │ ├── ingress.yaml # Ingress route
|
||||
│ │ ├── networkpolicy.yaml
|
||||
│ │ └── authentik-secret.yaml # Secrets (admin password, OIDC client secret, etc.)
|
||||
│ └── monitoring/ # Observability (Alloy, kube-state-metrics, node-exporter)
|
||||
│ ├── alloy-config.yaml # Grafana Alloy configuration
|
||||
│ └── kube-state-metrics.yaml # K8s metrics exporter
|
||||
├── host-config/ # Host-level (non-GitOps) config, see below
|
||||
│ └── maintenance-notify/ # systemd timer: pre-update mail/Matrix notifications (Issue #24)
|
||||
├── .sops.yaml # SOPS encryption rules (age key definition)
|
||||
├── scripts/
|
||||
│ ├── install-hooks.sh # Installs git hooks for ConfigMap auto-tracking
|
||||
│ └── hooks/ # Git hooks (pre-commit, post-commit, etc.)
|
||||
└── docs/
|
||||
├── README.md # Main deployment guide
|
||||
├── TASKS.md # Task list & milestones (backlog itself lives in Gitea issues)
|
||||
├── install.md # Installation instructions
|
||||
├── ops-configmap-sync.md # ConfigMap syncing with git hooks
|
||||
└── deployment-guides/ # Detailed guides for specific components (01-07)
|
||||
```
|
||||
|
||||
## Host-Level (non-GitOps) Changes
|
||||
|
||||
Almost everything in this repo is reconciled by Flux. `host-config/` is the deliberate
|
||||
exception: it holds scripts/systemd units meant to run **on the bare Hetzner host itself**
|
||||
(not as a Kubernetes pod), for things Flux structurally can't reach — e.g. host package
|
||||
management. There is no SOPS-on-host or Ansible-equivalent mechanism yet; deployment to the
|
||||
host is manual (`scp`/SSH), and instance-specific values live in a config file on the host
|
||||
(`/etc/<name>/config`), not hardcoded in the versioned script, so the pattern is reusable
|
||||
across forks/other communities running this same stack. See
|
||||
`docs/deployment-guides/07-host-maintenance-notifications.md` for the first (and so far only)
|
||||
example of this pattern.
|
||||
|
||||
## Common Development Commands
|
||||
|
||||
### Flux / GitOps Synchronization
|
||||
|
||||
```bash
|
||||
# Force immediate reconciliation (don't wait for 10-min auto-sync)
|
||||
flux reconcile kustomization flux-system --with-source
|
||||
flux reconcile kustomization production-apps --with-source
|
||||
|
||||
# Check reconciliation status
|
||||
flux get kustomizations -A
|
||||
flux get helmreleases -A
|
||||
|
||||
# View Flux logs
|
||||
kubectl logs -n flux-system deployment/source-controller -f
|
||||
kubectl logs -n flux-system deployment/helm-controller -f
|
||||
```
|
||||
|
||||
### Kubernetes Cluster Status
|
||||
|
||||
```bash
|
||||
# Check pod health in Matrix namespace
|
||||
kubectl get pods -n matrix
|
||||
kubectl get pods -n authentik
|
||||
kubectl get pods -n monitoring
|
||||
|
||||
# Detailed pod inspection
|
||||
kubectl describe pod <pod-name> -n matrix
|
||||
kubectl logs <pod-name> -n matrix -f
|
||||
|
||||
# Check all services and ingresses
|
||||
kubectl get svc -n matrix
|
||||
kubectl get ingress -n matrix
|
||||
```
|
||||
|
||||
### Certificate Management (Let's Encrypt / Cert-Manager)
|
||||
|
||||
```bash
|
||||
# View certificate status
|
||||
kubectl get certificate -n matrix
|
||||
kubectl get certificaterequest -n matrix
|
||||
kubectl get challenges -n matrix
|
||||
|
||||
# Debug failed certificate issuance
|
||||
kubectl describe challenge <challenge-name> -n matrix
|
||||
kubectl logs -n cert-manager deployment/cert-manager -f
|
||||
|
||||
# Inspect the issued certificate
|
||||
kubectl get secret <cert-secret-name> -n matrix -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout
|
||||
```
|
||||
|
||||
### SOPS Secret Editing
|
||||
|
||||
SOPS transparently encrypts/decrypts secrets using the `age` key specified in `.sops.yaml`. The environment variable `SOPS_AGE_KEY_FILE` must point to your age private key.
|
||||
|
||||
```bash
|
||||
# Edit an encrypted secret (decrypted for editing, re-encrypted on save)
|
||||
sops apps/production/custom-configs/mas-secret.yaml
|
||||
sops apps/authentik/authentik-secret.yaml
|
||||
|
||||
# Create a new secret file
|
||||
sops -i --encrypted-regex '^(data|stringData)$' --input-type yaml --output-type yaml new-secret.yaml
|
||||
|
||||
# Decrypt to view
|
||||
sops -d apps/production/custom-configs/mas-secret.yaml
|
||||
```
|
||||
|
||||
Ensure `~/.age/keys.txt` exists and contains your age private key. See `.devcontainer/devcontainer.json` for setup details.
|
||||
|
||||
### Helm Chart Inspection
|
||||
|
||||
```bash
|
||||
# List installed charts
|
||||
helm list -n matrix
|
||||
helm list -n authentik
|
||||
|
||||
# View rendered chart values
|
||||
helm get values matrix-stack -n matrix
|
||||
helm get manifest matrix-stack -n matrix | less
|
||||
```
|
||||
|
||||
### Useful kubectl Shortcuts
|
||||
|
||||
```bash
|
||||
# Port-forward to access services locally
|
||||
kubectl port-forward -n matrix svc/synapse 8008:8008
|
||||
|
||||
# Execute command inside pod (for debugging)
|
||||
kubectl exec -it <pod-name> -n matrix -- bash
|
||||
|
||||
# Stream logs from multiple pods
|
||||
kubectl logs -n matrix -l app=synapse -f
|
||||
|
||||
# Bootstrap a service/bot account via MAS (no registration_shared_secret in this stack)
|
||||
kubectl exec -it -n matrix deploy/matrix-stack-matrix-authentication-service -- \
|
||||
mas-cli manage register-user <name> --yes
|
||||
kubectl exec -it -n matrix deploy/matrix-stack-matrix-authentication-service -- \
|
||||
mas-cli manage issue-compatibility-token <name>
|
||||
```
|
||||
|
||||
## Architecture & Key Concepts
|
||||
|
||||
### FluxCD Reconciliation Flow
|
||||
|
||||
1. **Flux watches** `clusters/matrix/` for a FluxRepository resource pointing to this Git repo
|
||||
2. **Kustomization stages** pull in configurations in order:
|
||||
- `flux-system` (FluxCD itself)
|
||||
- `infra-apps` (Namespaces, RBAC, Cert-Manager, HelmRepository sources)
|
||||
- `production-apps` (Main ESS deployment and related services)
|
||||
3. **HelmReleases** specify which charts to install and what values to use
|
||||
4. **ConfigMaps/Secrets** provide values from files in the repo (e.g., custom Synapse config)
|
||||
5. **Flux auto-reconciles** every 10 minutes, or immediately if Git changes are detected
|
||||
|
||||
### Element Server Suite (ESS) Chart Constraints
|
||||
|
||||
The ESS Helm chart (v26.4.0) has strict validation and specific quirks:
|
||||
|
||||
- **No `config:` blocks for core components** — use ConfigMap overrides instead
|
||||
- **`serverName` must be at root level**, not nested under `synapse`
|
||||
- **TLS in Ingress blocks is forbidden** — use `certManager: true` at root to auto-manage certificates
|
||||
- **`camelCase` for component names**: `elementWeb`, `synapseAdmin`, `matrixAuthenticationService`, etc.
|
||||
- **OCI HelmRepository only** — the chart is distributed via `oci://ghcr.io/element-hq/ess-helm`, not HTTP
|
||||
- **Values must pass JSON schema validation** — invalid configs will cause reconciliation failures with cryptic schema errors
|
||||
|
||||
### NetworkPolicy Convention
|
||||
|
||||
Default-deny-with-explicit-allow across `matrix` and `authentik` namespaces
|
||||
(`apps/production/networkpolicy.yaml`, `apps/authentik/networkpolicy.yaml`). Every new pod
|
||||
needs its own explicit ingress-allow rule; NetworkPolicy matches on named **container ports**,
|
||||
not Service ports — a frequent source of live incidents when a new component is added (wrong
|
||||
port number/name silently blocks all traffic to it).
|
||||
|
||||
### Known Issues & Workarounds
|
||||
|
||||
**Issue: Let's Encrypt ACME Race Condition (Error 403 Order's status is processing)**
|
||||
- Symptom: Certificate provisioning hangs when `elementWeb` and `wellKnownDelegation` are both enabled on the same domain
|
||||
- Cause: Both request certificates for the same domain simultaneously; Let's Encrypt rejects concurrent requests
|
||||
- Fix: Set `wellKnownDelegation: enabled: false` and serve `.well-known/matrix/server` via a separate Ingress route or static file
|
||||
|
||||
**Issue: HelmChart not ready / stat no such file or directory**
|
||||
- Cause: Attempting to use a GitRepository source for the ESS chart (it has sub-charts that don't render correctly)
|
||||
- Fix: Use the OCI HelmRepository source (`oci://ghcr.io/element-hq/ess-helm`) instead
|
||||
|
||||
**Issue: Certificate validation failures (No resources found)**
|
||||
- Cause: Manual Kustomize patches conflict with the Helm chart's built-in certificate management
|
||||
- Fix: Remove manual patches; rely on `certManager: true` at the root level of HelmRelease values
|
||||
|
||||
**Issue: Synapse module can't use asyncio**
|
||||
- Cause: Synapse runs on Twisted's reactor, not a running asyncio event loop — `asyncio.open_connection`/`asyncio.wait_for` inside a Synapse module (e.g. `clamav_spam_checker.py`) fail immediately with `RuntimeError: no running event loop`, and can silently trigger a fail-open path instead of an obvious crash
|
||||
- Fix: use `twisted.internet.reactor`/`HostnameEndpoint`/`connectProtocol` + a custom `Protocol` subclass; Twisted `Deferred`s are natively awaitable from `async def` inside Synapse. Standalone processes outside Synapse (e.g. `clamav-http-scanner.py`) don't have this constraint and can use plain sockets/asyncio.
|
||||
|
||||
### SOPS Encryption & Key Management
|
||||
|
||||
- `.sops.yaml` defines encryption rules (currently using `age` keys)
|
||||
- Secrets matching the regex in `.sops.yaml` are automatically encrypted when committed
|
||||
- The age private key (`~/.age/keys.txt`) must be available in your environment for decryption
|
||||
- In the cluster, Flux decrypts secrets "on the fly" using a secret stored in `flux-system` namespace
|
||||
|
||||
To rotate SOPS keys:
|
||||
```bash
|
||||
# Regenerate and re-encrypt all secrets
|
||||
sops updatekeys -y apps/
|
||||
```
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Before Making Changes
|
||||
|
||||
1. **Understand dependencies** — check `kustomization.yaml` files to see the order of resource creation
|
||||
2. **Verify chart schema** — review ESS chart documentation for constraints on the version being used
|
||||
3. **Test locally if possible** — use `kubectl` port-forwards to verify connectivity before pushing changes
|
||||
|
||||
### Making Changes
|
||||
|
||||
1. **Edit ConfigMap files directly** — for non-secret customizations (Synapse config, Element Web themes, etc.)
|
||||
- Changes are auto-tracked by git hooks installed via `./scripts/install-hooks.sh`
|
||||
2. **Edit secrets with SOPS** — `sops` transparently decrypts/re-encrypts on save
|
||||
3. **Update HelmRelease values** — modify the `values` section in `element-server-suite.yaml` or reference ConfigMap sources
|
||||
|
||||
### After Committing
|
||||
|
||||
1. **Flux auto-detects changes** within ~1 minute (or manually trigger with `flux reconcile kustomization production-apps`)
|
||||
2. **Monitor reconciliation** — watch pod logs and Flux status for errors
|
||||
3. **Test functionality** — verify services are accessible and functioning as expected
|
||||
|
||||
### Git Hooks
|
||||
|
||||
After cloning, run:
|
||||
```bash
|
||||
./scripts/install-hooks.sh
|
||||
```
|
||||
|
||||
This installs hooks that automatically commit ConfigMap changes to `.gitignore`-like tracking. See `docs/ops-configmap-sync.md` for details.
|
||||
|
||||
## Environment Setup
|
||||
|
||||
### Local Machine Prerequisites
|
||||
|
||||
- `kubectl` — cluster communication
|
||||
- `flux` — GitOps CLI
|
||||
- `helm` — chart inspection & debugging
|
||||
- `sops` & `age` — secret management
|
||||
- `git` — version control
|
||||
- age key file at `~/.age/keys.txt` (request from team)
|
||||
- kubeconfig at `~/.kube/config` (request from team)
|
||||
|
||||
### DevContainer (Recommended)
|
||||
|
||||
The `.devcontainer/` configuration provides a pre-configured environment:
|
||||
|
||||
```bash
|
||||
# In VS Code: "Reopen in Container"
|
||||
# Or manually:
|
||||
docker build -t ess-devcontainer .devcontainer
|
||||
docker run -it --rm \
|
||||
-v ~/.kube:/home/vscode/.kube \
|
||||
-v ~/.age:/home/vscode/.age \
|
||||
-v ~/.ssh:/home/vscode/.ssh \
|
||||
-v /var/run/docker.sock:/var/run/docker.sock \
|
||||
ess-devcontainer
|
||||
```
|
||||
|
||||
DevContainer includes:
|
||||
- All required CLI tools (kubectl, flux, helm, sops, age, git, docker)
|
||||
- VS Code extensions for YAML, Kubernetes, Helm
|
||||
- Proper environment variables (`KUBECONFIG`, `SOPS_AGE_KEY_FILE`)
|
||||
- Git hooks pre-installed
|
||||
|
||||
## Troubleshooting Checklist
|
||||
|
||||
- **Pod not starting?** → `kubectl describe pod <name> -n matrix` (check events)
|
||||
- **Image pull failures?** → Check HelmRelease status: `kubectl get helmrelease -n matrix`
|
||||
- **Secret not found?** → Verify SOPS decryption: `sops -d <secret.sops.yaml>` (must output valid YAML)
|
||||
- **Certificate stuck?** → `kubectl describe certificate <name> -n matrix` (check for ACME errors)
|
||||
- **Config validation error?** → Inspect HelmRelease status: `kubectl describe helmrelease <name> -n matrix` (JSON schema error message)
|
||||
- **Cluster unreachable?** → Verify kubeconfig: `kubectl get nodes` (must connect to K3s)
|
||||
- **NetworkPolicy blocking a new pod?** → Check it matches on container port name, not Service port
|
||||
|
||||
## Resources & References
|
||||
|
||||
- **README.md** — High-level overview and architecture
|
||||
- **docs/TASKS.md** — Task backlog, milestones, and priority list (open backlog lives in [Gitea issues](https://rohana.axion1337.de/sorb/axion1337.chat-gitops/issues))
|
||||
- **docs/deployment-guides/** — Detailed setup guides for specific components (01-07)
|
||||
- **docs/ops-configmap-sync.md** — Git hook configuration and auto-sync behavior
|
||||
- **ESS Chart Docs** — `https://github.com/element-hq/ess-helm` (official Helm chart repository)
|
||||
- **FluxCD Docs** — `https://fluxcd.io/docs/` (GitOps reconciliation & Kustomization)
|
||||
- **Matrix Spec** — `https://spec.matrix.org/` (Matrix protocol specification)
|
||||
Reference in New Issue
Block a user