From 23533c7899da20dd35ba25339714d0629c9ee32e Mon Sep 17 00:00:00 2001 From: Thore Cimbal Date: Sat, 15 Aug 2026 12:00:00 +0000 Subject: [PATCH] docs: split CLAUDE.md into a pointer plus AGENTS.md Follows the management repo's shape (ADR-0013): CLAUDE.md is the one-line pointer the group check looks for, AGENTS.md carries the project specifics unchanged. The group-rules reference now points at management's AGENTS.md rather than its CLAUDE.md, which had itself become a pointer. Co-Authored-By: Claude Opus 4.8 --- AGENTS.md | 464 +++++++++++++++++++++++++++++++++++++++++++++++++++++ CLAUDE.md | 467 +----------------------------------------------------- 2 files changed, 465 insertions(+), 466 deletions(-) create mode 100644 AGENTS.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..ee06b20 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,464 @@ +# AGENTS.md — axion1337.chat-gitops + +> **Die Gruppenregeln sind kanonisch im `management`-Repo:** +> [`AGENTS.md`](https://git.lab/axion1337.chat/management/-/blob/main/AGENTS.md) +> — von außerhalb des Labs über den Gitea-Mirror lesbar: +> `https://rohana.axion1337.de/sorb/management`. Dort stehen Repo-Topologie und +> Mirror-Regeln, das Kanban-Framework (Status-Labels, WIP-Limit 2, ADR-Pflicht — +> auch für dauerhafte Regel-Ausnahmen), Deploy-Übergabe und AAR-Verfahren, +> Secrets-Handhabung und die Karpathy-Leitlinien. Sie gelten für **jede** Session +> in diesem Repo. Hier steht nur, was für dieses Repository zusätzlich gilt. + +## Overview + +This is a **GitOps-based Kubernetes deployment** of **Element Server Suite (ESS Community v26.4.0)**, a complete Matrix homeserver stack. The repository contains Infrastructure-as-Code using **FluxCD** for GitOps synchronization, with encryption (SOPS/age), service mesh (Traefik), certificate management (Cert-Manager), and auxiliary services like Authentik, TURN/coturn, Draupnir (moderation), ClamAV (content scanning), and Grafana monitoring. + +**Key Stack Components:** +- **K3s**: Lightweight Kubernetes distribution running on Hetzner Cloud +- **FluxCD**: GitOps controller that watches this repository and auto-syncs changes +- **ESS (Matrix Umbrella Chart v26.4.0)**: Synapse, MAS (Matrix Authentication Service), ElementWeb, MatrixRTC +- **Authentik**: OIDC-based identity provider for centralized authentication, deployed both via HelmRelease and declarative Blueprints (`apps/authentik/authentik-blueprints.yaml`) for flows/OIDC-provider config that would otherwise only exist as manual admin-UI clicks +- **Traefik**: Ingress controller (built into K3s) for routing HTTP/HTTPS traffic +- **Cert-Manager**: Automatic TLS certificate provisioning from Let's Encrypt +- **SOPS + age**: Transparent encryption/decryption of secrets in Git +- **Monitoring**: Grafana Alloy (agent), Prometheus (metrics), Loki (logs) +- **coturn**: TURN/STUN server for WebRTC audio/video calls, with monthly automated shared-secret rotation via CronJob + PR workflow +- **Draupnir**: Matrix moderation bot (community successor to Mjolnir), ban lists/policy rooms +- **ClamAV**: Content scanning — a Synapse module for unencrypted-room uploads, plus a standalone `clamav-http-scanner` service that a patched Element Web client (ThreadNet-Web) calls both on send and on receive, extending coverage to encrypted rooms/DMs +- **NetworkPolicies**: default-deny-with-explicit-allow across `matrix` and `authentik` namespaces +- **`host-config/`**: the one part of this repo that is deliberately **not** managed by Flux/GitOps — see "Host-Level (non-GitOps) Changes" below + +## Repo Topology (since 2026-07-31) + +Canonical repo is **`git.lab/axion1337.chat/axion1337.chat-gitops`** (homelab GitLab, +resolvable only inside the lab) — all pushes go there; a push-mirror updates the Gitea +copy on `rohana.axion1337.de`, which remains the **Flux source** (the cluster pulls from +Gitea; the mirror delivers). **Never push directly to Gitea** for this repo — the mirror +force-overwrites divergent state. The same rule applies to ThreadNet-Web, threadnet-call, +thread-net-git, threadnet-operating and (since 2026-08-01) `management` (the former +`Backlogs` repo, renamed in the PM-framework restructuring — ADRs/vision/roadmap live +there). Since 2026-08-02 the wiki lives on git.lab too, so **no repo is authored on +Gitea any more**; the one process that still *writes* there is the TURN rotation +(see below). + +**Why Gitea is the Flux source, and why that is not a leftover.** git.lab holds the +blueprints; Gitea holds a copy the cluster can reach without the lab being up. That +separation is deliberate: the Hetzner cluster must be buildable and re-deployable when +the homelab is offline, on holiday, or mid-rebuild — it therefore must not depend on a +host that only answers inside the lab. **Do not "fix" the Flux source to point at +git.lab**: it would look tidier and would couple production availability to the lab, +which is exactly what this split avoids. + +**Issues live on git.lab** (migrated 2026-08-01, gitops#48): the old Gitea issues are +closed with a pointer to their GitLab counterpart. ⚠️ gitops issue numbers **shifted** +in the migration (Gitea had gaps from PRs; e.g. Gitea#48 → GitLab#46) — old references +"gitops#N" in commits/docs mean the **Gitea** number; the authoritative mapping is the +migration footer in each GitLab issue. Releases stay on Gitea (public download path). +**No exceptions left**: deploy-handover issues used to stay on the Gitea +`sorb/management` tracker because CFGMON had no route to git.lab. The site-to-site VPN +(ADR-0004) removed that reason, and on 2026-08-02 they moved (LABNET-03) — the Gitea +tracker is empty and the form now lives as a GitLab issue template in the management +repo. Issues live on git.lab, without exception. + +**Wiki** (migrated 2026-08-02): the 15 pages now live in this project's GitLab wiki +(`…-gitops.wiki.git`, *Wiki* tab). ⚠️ The **`wiki` branch in this repo is NOT the wiki** — +it is a stale May snapshot of `docs/`; don't edit or trust it. All doc sources +(platform wiki, `homelab/docs`, `management`) are also served side by side as one +Docusaurus site at **wiki.lab**, configured in `git.lab/homelab/wiki` — content is +pulled at build time, so edits always belong in the source repo. + +**The one write that still lands on Gitea**: the monthly TURN-rotation CronJob runs in +the cluster (no route to git.lab) and pushes its rotation branch to Gitea. Never merge +that PR on Gitea — the rotation has to travel back through git.lab, otherwise the next +mirror push overwrites it and Flux silently re-applies the *old* shared secret. + +**This is automated — do not do it by hand.** The scheduled job `canonize_rotation` in +`.gitlab-ci.yml` runs daily on git.lab, picks up any `turn-secret-rotation-*` branch +from Gitea that is not yet in `main`, merges it, and pushes to git.lab; the mirror +carries it back and Gitea auto-closes the PR. Once merged, the branch exists only on +Gitea, so the next mirror run deletes it — no cleanup needed. + +If the job fails (merge conflict, or the SOPS check finds an unencrypted secret file), +it fails **loudly and changes nothing** — the pipeline stays red until someone looks. +That red pipeline is the alarm; there is no separate reminder. Manual fallback, should +it ever be needed: fetch the branch, merge into main locally, push to git.lab (worked +example: 2026-08-01, commit `717638d`). If the mirror lags, force it via the GitLab API +`POST /projects//remote_mirrors//sync`. + +## Repository Structure + +``` +gitops/ +├── .gitlab-ci.yml # Lab-CI verify job (YAML validation + SOPS-encryption check) +├── clusters/matrix/ # Flux GitRepository definition; entry point for reconciliation +├── apps/ +│ ├── base/ +│ │ ├── infra/ # Core infrastructure (Cert-Manager, Namespaces, etc.) +│ │ └── matrix/ # HelmRepository definition for ESS OCI chart +│ ├── production/ # Main ESS deployment +│ │ ├── element-server-suite.yaml # HelmRelease (ESS chart v26.4.0) +│ │ ├── custom-configs/ # Overrides & custom configurations +│ │ │ ├── synapse-values.yaml # Synapse customizations (ConfigMap) +│ │ │ ├── element-values.yaml # ElementWeb customizations (ConfigMap) +│ │ │ └── mas-secret.yaml # MAS secrets (encrypted with SOPS) +│ │ ├── cert-issuer.yaml # Let's Encrypt ClusterIssuer +│ │ ├── apex-ingress.yaml # Apex-domain IngressRoutes (Element Web, /_scan, etc.) +│ │ ├── matrix-postgres-auth.yaml # PostgreSQL credentials +│ │ ├── coturn.yaml / coturn-secret.yaml / synapse-turn-secret.yaml +│ │ ├── turn-secret-rotation.yaml # Monthly CronJob, rotates coturn shared secret via PR +│ │ ├── draupnir.yaml / draupnir-pvc.yaml / draupnir-secret.yaml +│ │ ├── clamav.yaml / clamav-pvc.yaml / clamav_spam_checker.py # Synapse-side scan module +│ │ ├── clamav-http-scanner.py / -Dockerfile / .yaml # Client-side scan service +│ │ ├── synapse-backup.yaml / synapse-backup-secret.yaml +│ │ └── networkpolicy.yaml # Default-deny + explicit allow rules +│ ├── authentik/ # Identity Provider (separate namespace) +│ │ ├── authentik.yaml # HelmRelease +│ │ ├── authentik-blueprints.yaml # Flows/OIDC-provider as declarative code +│ │ ├── helm-repo.yaml # HelmRepository source +│ │ ├── ingress.yaml # Ingress route +│ │ ├── networkpolicy.yaml +│ │ └── authentik-secret.yaml # Secrets (admin password, OIDC client secret, etc.) +│ └── monitoring/ # Observability (Alloy, kube-state-metrics, node-exporter) +│ ├── alloy-config.yaml # Grafana Alloy configuration +│ └── kube-state-metrics.yaml # K8s metrics exporter +├── host-config/ # Host-level (non-GitOps) config, see below +│ └── maintenance-notify/ # systemd timer: pre-update mail/Matrix notifications (Issue #24) +├── .sops.yaml # SOPS encryption rules (age key definition) +├── scripts/ +│ ├── install-hooks.sh # Installs git hooks for ConfigMap auto-tracking +│ └── hooks/ # Git hooks (pre-commit, post-commit, etc.) +└── docs/ + ├── README.md # Main deployment guide + ├── TASKS.md # Task list & milestones (backlog itself lives in git.lab issues) + ├── install.md # Installation instructions + ├── ops-configmap-sync.md # ConfigMap syncing with git hooks + └── deployment-guides/ # Detailed guides for specific components (01-07) +``` + +## Host-Level (non-GitOps) Changes + +Almost everything in this repo is reconciled by Flux. `host-config/` is the deliberate +exception: it holds scripts/systemd units meant to run **on the bare Hetzner host itself** +(not as a Kubernetes pod), for things Flux structurally can't reach — e.g. host package +management. There is no SOPS-on-host or Ansible-equivalent mechanism yet; deployment to the +host is manual (`scp`/SSH), and instance-specific values live in a config file on the host +(`/etc//config`), not hardcoded in the versioned script, so the pattern is reusable +across forks/other communities running this same stack. See +`docs/deployment-guides/07-host-maintenance-notifications.md` for the first (and so far only) +example of this pattern. + +## Common Development Commands + +### Flux / GitOps Synchronization + +```bash +# Force immediate reconciliation (don't wait for 10-min auto-sync) +flux reconcile kustomization flux-system --with-source +flux reconcile kustomization production-apps --with-source + +# Check reconciliation status +flux get kustomizations -A +flux get helmreleases -A + +# View Flux logs +kubectl logs -n flux-system deployment/source-controller -f +kubectl logs -n flux-system deployment/helm-controller -f +``` + +### Kubernetes Cluster Status + +```bash +# Check pod health in Matrix namespace +kubectl get pods -n matrix +kubectl get pods -n authentik +kubectl get pods -n monitoring + +# Detailed pod inspection +kubectl describe pod -n matrix +kubectl logs -n matrix -f + +# Check all services and ingresses +kubectl get svc -n matrix +kubectl get ingress -n matrix +``` + +### Certificate Management (Let's Encrypt / Cert-Manager) + +```bash +# View certificate status +kubectl get certificate -n matrix +kubectl get certificaterequest -n matrix +kubectl get challenges -n matrix + +# Debug failed certificate issuance +kubectl describe challenge -n matrix +kubectl logs -n cert-manager deployment/cert-manager -f + +# Inspect the issued certificate +kubectl get secret -n matrix -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout +``` + +### SOPS Secret Editing + +SOPS transparently encrypts/decrypts secrets using the `age` key specified in `.sops.yaml`. The environment variable `SOPS_AGE_KEY_FILE` must point to your age private key. + +```bash +# Edit an encrypted secret (decrypted for editing, re-encrypted on save) +sops apps/production/custom-configs/mas-secret.yaml +sops apps/authentik/authentik-secret.yaml + +# Create a new secret file +sops -i --encrypted-regex '^(data|stringData)$' --input-type yaml --output-type yaml new-secret.yaml + +# Decrypt to view +sops -d apps/production/custom-configs/mas-secret.yaml +``` + +Ensure `~/.age/keys.txt` exists and contains your age private key. See `.devcontainer/devcontainer.json` for setup details. + +### Helm Chart Inspection + +```bash +# List installed charts +helm list -n matrix +helm list -n authentik + +# View rendered chart values +helm get values matrix-stack -n matrix +helm get manifest matrix-stack -n matrix | less +``` + +### Useful kubectl Shortcuts + +```bash +# Port-forward to access services locally +kubectl port-forward -n matrix svc/synapse 8008:8008 + +# Execute command inside pod (for debugging) +kubectl exec -it -n matrix -- bash + +# Stream logs from multiple pods +kubectl logs -n matrix -l app=synapse -f + +# Bootstrap a service/bot account via MAS (no registration_shared_secret in this stack) +kubectl exec -it -n matrix deploy/matrix-stack-matrix-authentication-service -- \ + mas-cli manage register-user --yes +kubectl exec -it -n matrix deploy/matrix-stack-matrix-authentication-service -- \ + mas-cli manage issue-compatibility-token +``` + +## Architecture & Key Concepts + +### FluxCD Reconciliation Flow + +1. **Flux watches** `clusters/matrix/` for a FluxRepository resource pointing to this Git repo +2. **Kustomization stages** pull in configurations in order: + - `flux-system` (FluxCD itself) + - `infra-apps` (Namespaces, RBAC, Cert-Manager, HelmRepository sources) + - `production-apps` (Main ESS deployment and related services) +3. **HelmReleases** specify which charts to install and what values to use +4. **ConfigMaps/Secrets** provide values from files in the repo (e.g., custom Synapse config) +5. **Flux auto-reconciles** every 10 minutes, or immediately if Git changes are detected + +### Element Server Suite (ESS) Chart Constraints + +The ESS Helm chart (v26.4.0) has strict validation and specific quirks: + +- **No `config:` blocks for core components** — use ConfigMap overrides instead +- **`serverName` must be at root level**, not nested under `synapse` +- **TLS in Ingress blocks is forbidden** — use `certManager: true` at root to auto-manage certificates +- **`camelCase` for component names**: `elementWeb`, `synapseAdmin`, `matrixAuthenticationService`, etc. +- **OCI HelmRepository only** — the chart is distributed via `oci://ghcr.io/element-hq/ess-helm`, not HTTP +- **Values must pass JSON schema validation** — invalid configs will cause reconciliation failures with cryptic schema errors + +### NetworkPolicy Convention + +Default-deny-with-explicit-allow across `matrix` and `authentik` namespaces +(`apps/production/networkpolicy.yaml`, `apps/authentik/networkpolicy.yaml`). Every new pod +needs its own explicit ingress-allow rule; NetworkPolicy matches on named **container ports**, +not Service ports — a frequent source of live incidents when a new component is added (wrong +port number/name silently blocks all traffic to it). + +### Known Issues & Workarounds + +**Issue: Let's Encrypt ACME Race Condition (Error 403 Order's status is processing)** +- Symptom: Certificate provisioning hangs when `elementWeb` and `wellKnownDelegation` are both enabled on the same domain +- Cause: Both request certificates for the same domain simultaneously; Let's Encrypt rejects concurrent requests +- Fix: Set `wellKnownDelegation: enabled: false` and serve `.well-known/matrix/server` via a separate Ingress route or static file + +**Issue: HelmChart not ready / stat no such file or directory** +- Cause: Attempting to use a GitRepository source for the ESS chart (it has sub-charts that don't render correctly) +- Fix: Use the OCI HelmRepository source (`oci://ghcr.io/element-hq/ess-helm`) instead + +**Issue: Certificate validation failures (No resources found)** +- Cause: Manual Kustomize patches conflict with the Helm chart's built-in certificate management +- Fix: Remove manual patches; rely on `certManager: true` at the root level of HelmRelease values + +**Issue: Synapse module can't use asyncio** +- Cause: Synapse runs on Twisted's reactor, not a running asyncio event loop — `asyncio.open_connection`/`asyncio.wait_for` inside a Synapse module (e.g. `clamav_spam_checker.py`) fail immediately with `RuntimeError: no running event loop`, and can silently trigger a fail-open path instead of an obvious crash +- Fix: use `twisted.internet.reactor`/`HostnameEndpoint`/`connectProtocol` + a custom `Protocol` subclass; Twisted `Deferred`s are natively awaitable from `async def` inside Synapse. Standalone processes outside Synapse (e.g. `clamav-http-scanner.py`) don't have this constraint and can use plain sockets/asyncio. + +### SOPS Encryption & Key Management + +- `.sops.yaml` defines encryption rules (currently using `age` keys) +- Secrets matching the regex in `.sops.yaml` are automatically encrypted when committed +- The age private key (`~/.age/keys.txt`) must be available in your environment for decryption +- In the cluster, Flux decrypts secrets "on the fly" using a secret stored in `flux-system` namespace + +To rotate SOPS keys: +```bash +# Regenerate and re-encrypt all secrets +sops updatekeys -y apps/ +``` + +## Development Workflow + +### Before Making Changes + +1. **Understand dependencies** — check `kustomization.yaml` files to see the order of resource creation +2. **Verify chart schema** — review ESS chart documentation for constraints on the version being used +3. **Test locally if possible** — use `kubectl` port-forwards to verify connectivity before pushing changes + +### Making Changes + +1. **Edit ConfigMap files directly** — for non-secret customizations (Synapse config, Element Web themes, etc.) + - Changes are auto-tracked by git hooks installed via `./scripts/install-hooks.sh` +2. **Edit secrets with SOPS** — `sops` transparently decrypts/re-encrypts on save +3. **Update HelmRelease values** — modify the `values` section in `element-server-suite.yaml` or reference ConfigMap sources + +### After Committing + +1. **Flux auto-detects changes** within ~1 minute (or manually trigger with `flux reconcile kustomization production-apps`) +2. **Monitor reconciliation** — watch pod logs and Flux status for errors +3. **Test functionality** — verify services are accessible and functioning as expected + +### Git Hooks + +After cloning, run: +```bash +./scripts/install-hooks.sh +``` + +This installs hooks that automatically commit ConfigMap changes to `.gitignore`-like tracking. See `docs/ops-configmap-sync.md` for details. + +## Environment Setup + +### Local Machine Prerequisites + +- `kubectl` — cluster communication +- `flux` — GitOps CLI +- `helm` — chart inspection & debugging +- `sops` & `age` — secret management +- `git` — version control +- age key file at `~/.age/keys.txt` (request from team) +- kubeconfig at `~/.kube/config` (request from team) + +### DevContainer (Recommended) + +The `.devcontainer/` configuration provides a pre-configured environment: + +```bash +# In VS Code: "Reopen in Container" +# Or manually: +docker build -t ess-devcontainer .devcontainer +docker run -it --rm \ + -v ~/.kube:/home/vscode/.kube \ + -v ~/.age:/home/vscode/.age \ + -v ~/.ssh:/home/vscode/.ssh \ + -v /var/run/docker.sock:/var/run/docker.sock \ + ess-devcontainer +``` + +DevContainer includes: +- All required CLI tools (kubectl, flux, helm, sops, age, git, docker) +- VS Code extensions for YAML, Kubernetes, Helm +- Proper environment variables (`KUBECONFIG`, `SOPS_AGE_KEY_FILE`) +- Git hooks pre-installed + +## Troubleshooting Checklist + +- **Pod not starting?** → `kubectl describe pod -n matrix` (check events) +- **Image pull failures?** → Check HelmRelease status: `kubectl get helmrelease -n matrix` +- **Secret not found?** → Verify SOPS decryption: `sops -d ` (must output valid YAML) +- **Certificate stuck?** → `kubectl describe certificate -n matrix` (check for ACME errors) +- **Config validation error?** → Inspect HelmRelease status: `kubectl describe helmrelease -n matrix` (JSON schema error message) +- **Cluster unreachable?** → Verify kubeconfig: `kubectl get nodes` (must connect to K3s) +- **NetworkPolicy blocking a new pod?** → Check it matches on container port name, not Service port + +## Resources & References + +- **README.md** — High-level overview and architecture +- **docs/TASKS.md** — Task backlog, milestones, and priority list (open backlog lives in [git.lab issues](https://git.lab/axion1337.chat/axion1337.chat-gitops/-/issues); the closed Gitea issues only point there) +- **docs/deployment-guides/** — Detailed setup guides for specific components (01-07) +- **docs/ops-configmap-sync.md** — Git hook configuration and auto-sync behavior +- **ESS Chart Docs** — `https://github.com/element-hq/ess-helm` (official Helm chart repository) +- **FluxCD Docs** — `https://fluxcd.io/docs/` (GitOps reconciliation & Kustomization) +- **Matrix Spec** — `https://spec.matrix.org/` (Matrix protocol specification) + +--- +name: karpathy-guidelines +description: Behavioral guidelines to reduce common LLM coding mistakes. Use when writing, reviewing, or refactoring code to avoid overcomplication, make surgical changes, surface assumptions, and define verifiable success criteria. +license: MIT +--- + +# Karpathy Guidelines + +Behavioral guidelines to reduce common LLM coding mistakes, derived from [Andrej Karpathy's observations](https://x.com/karpathy/status/2015883857489522876) on LLM coding pitfalls. + +**Tradeoff:** These guidelines bias toward caution over speed. For trivial tasks, use judgment. + +## 1. Think Before Coding + +**Don't assume. Don't hide confusion. Surface tradeoffs.** + +Before implementing: +- State your assumptions explicitly. If uncertain, ask. +- If multiple interpretations exist, present them - don't pick silently. +- If a simpler approach exists, say so. Push back when warranted. +- If something is unclear, stop. Name what's confusing. Ask. + +## 2. Simplicity First + +**Minimum code that solves the problem. Nothing speculative.** + +- No features beyond what was asked. +- No abstractions for single-use code. +- No "flexibility" or "configurability" that wasn't requested. +- No error handling for impossible scenarios. +- If you write 200 lines and it could be 50, rewrite it. + +Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify. + +## 3. Surgical Changes + +**Touch only what you must. Clean up only your own mess.** + +When editing existing code: +- Don't "improve" adjacent code, comments, or formatting. +- Don't refactor things that aren't broken. +- Match existing style, even if you'd do it differently. +- If you notice unrelated dead code, mention it - don't delete it. + +When your changes create orphans: +- Remove imports/variables/functions that YOUR changes made unused. +- Don't remove pre-existing dead code unless asked. + +The test: Every changed line should trace directly to the user's request. + +## 4. Goal-Driven Execution + +**Define success criteria. Loop until verified.** + +Transform tasks into verifiable goals: +- "Add validation" → "Write tests for invalid inputs, then make them pass" +- "Fix the bug" → "Write a test that reproduces it, then make it pass" +- "Refactor X" → "Ensure tests pass before and after" + +For multi-step tasks, state a brief plan: +``` +1. [Step] → verify: [check] +2. [Step] → verify: [check] +3. [Step] → verify: [check] +``` + +Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification. diff --git a/CLAUDE.md b/CLAUDE.md index f5a2f41..9aef12f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,466 +1 @@ -# CLAUDE.md - -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. - -> **Cross-project working conventions are canonical in the `management` repo's -> [`CLAUDE.md`](https://git.lab/axion1337.chat/management/-/blob/main/CLAUDE.md)** -> (readable from anywhere via the Gitea mirror -> `https://rohana.axion1337.de/sorb/management`): repo topology & mirror rules, -> the Kanban framework (status labels, WIP limit 2, ADR duty — including for -> permanent rule exceptions), deploy-handover/AAR procedures, secrets handling, -> and the Karpathy guidelines. Every agent session must follow it; this file -> covers only what is specific to this repository. - -## Overview - -This is a **GitOps-based Kubernetes deployment** of **Element Server Suite (ESS Community v26.4.0)**, a complete Matrix homeserver stack. The repository contains Infrastructure-as-Code using **FluxCD** for GitOps synchronization, with encryption (SOPS/age), service mesh (Traefik), certificate management (Cert-Manager), and auxiliary services like Authentik, TURN/coturn, Draupnir (moderation), ClamAV (content scanning), and Grafana monitoring. - -**Key Stack Components:** -- **K3s**: Lightweight Kubernetes distribution running on Hetzner Cloud -- **FluxCD**: GitOps controller that watches this repository and auto-syncs changes -- **ESS (Matrix Umbrella Chart v26.4.0)**: Synapse, MAS (Matrix Authentication Service), ElementWeb, MatrixRTC -- **Authentik**: OIDC-based identity provider for centralized authentication, deployed both via HelmRelease and declarative Blueprints (`apps/authentik/authentik-blueprints.yaml`) for flows/OIDC-provider config that would otherwise only exist as manual admin-UI clicks -- **Traefik**: Ingress controller (built into K3s) for routing HTTP/HTTPS traffic -- **Cert-Manager**: Automatic TLS certificate provisioning from Let's Encrypt -- **SOPS + age**: Transparent encryption/decryption of secrets in Git -- **Monitoring**: Grafana Alloy (agent), Prometheus (metrics), Loki (logs) -- **coturn**: TURN/STUN server for WebRTC audio/video calls, with monthly automated shared-secret rotation via CronJob + PR workflow -- **Draupnir**: Matrix moderation bot (community successor to Mjolnir), ban lists/policy rooms -- **ClamAV**: Content scanning — a Synapse module for unencrypted-room uploads, plus a standalone `clamav-http-scanner` service that a patched Element Web client (ThreadNet-Web) calls both on send and on receive, extending coverage to encrypted rooms/DMs -- **NetworkPolicies**: default-deny-with-explicit-allow across `matrix` and `authentik` namespaces -- **`host-config/`**: the one part of this repo that is deliberately **not** managed by Flux/GitOps — see "Host-Level (non-GitOps) Changes" below - -## Repo Topology (since 2026-07-31) - -Canonical repo is **`git.lab/axion1337.chat/axion1337.chat-gitops`** (homelab GitLab, -resolvable only inside the lab) — all pushes go there; a push-mirror updates the Gitea -copy on `rohana.axion1337.de`, which remains the **Flux source** (the cluster pulls from -Gitea; the mirror delivers). **Never push directly to Gitea** for this repo — the mirror -force-overwrites divergent state. The same rule applies to ThreadNet-Web, threadnet-call, -thread-net-git, threadnet-operating and (since 2026-08-01) `management` (the former -`Backlogs` repo, renamed in the PM-framework restructuring — ADRs/vision/roadmap live -there). Since 2026-08-02 the wiki lives on git.lab too, so **no repo is authored on -Gitea any more**; the one process that still *writes* there is the TURN rotation -(see below). - -**Why Gitea is the Flux source, and why that is not a leftover.** git.lab holds the -blueprints; Gitea holds a copy the cluster can reach without the lab being up. That -separation is deliberate: the Hetzner cluster must be buildable and re-deployable when -the homelab is offline, on holiday, or mid-rebuild — it therefore must not depend on a -host that only answers inside the lab. **Do not "fix" the Flux source to point at -git.lab**: it would look tidier and would couple production availability to the lab, -which is exactly what this split avoids. - -**Issues live on git.lab** (migrated 2026-08-01, gitops#48): the old Gitea issues are -closed with a pointer to their GitLab counterpart. ⚠️ gitops issue numbers **shifted** -in the migration (Gitea had gaps from PRs; e.g. Gitea#48 → GitLab#46) — old references -"gitops#N" in commits/docs mean the **Gitea** number; the authoritative mapping is the -migration footer in each GitLab issue. Releases stay on Gitea (public download path). -**No exceptions left**: deploy-handover issues used to stay on the Gitea -`sorb/management` tracker because CFGMON had no route to git.lab. The site-to-site VPN -(ADR-0004) removed that reason, and on 2026-08-02 they moved (LABNET-03) — the Gitea -tracker is empty and the form now lives as a GitLab issue template in the management -repo. Issues live on git.lab, without exception. - -**Wiki** (migrated 2026-08-02): the 15 pages now live in this project's GitLab wiki -(`…-gitops.wiki.git`, *Wiki* tab). ⚠️ The **`wiki` branch in this repo is NOT the wiki** — -it is a stale May snapshot of `docs/`; don't edit or trust it. All doc sources -(platform wiki, `homelab/docs`, `management`) are also served side by side as one -Docusaurus site at **wiki.lab**, configured in `git.lab/homelab/wiki` — content is -pulled at build time, so edits always belong in the source repo. - -**The one write that still lands on Gitea**: the monthly TURN-rotation CronJob runs in -the cluster (no route to git.lab) and pushes its rotation branch to Gitea. Never merge -that PR on Gitea — the rotation has to travel back through git.lab, otherwise the next -mirror push overwrites it and Flux silently re-applies the *old* shared secret. - -**This is automated — do not do it by hand.** The scheduled job `canonize_rotation` in -`.gitlab-ci.yml` runs daily on git.lab, picks up any `turn-secret-rotation-*` branch -from Gitea that is not yet in `main`, merges it, and pushes to git.lab; the mirror -carries it back and Gitea auto-closes the PR. Once merged, the branch exists only on -Gitea, so the next mirror run deletes it — no cleanup needed. - -If the job fails (merge conflict, or the SOPS check finds an unencrypted secret file), -it fails **loudly and changes nothing** — the pipeline stays red until someone looks. -That red pipeline is the alarm; there is no separate reminder. Manual fallback, should -it ever be needed: fetch the branch, merge into main locally, push to git.lab (worked -example: 2026-08-01, commit `717638d`). If the mirror lags, force it via the GitLab API -`POST /projects//remote_mirrors//sync`. - -## Repository Structure - -``` -gitops/ -├── .gitlab-ci.yml # Lab-CI verify job (YAML validation + SOPS-encryption check) -├── clusters/matrix/ # Flux GitRepository definition; entry point for reconciliation -├── apps/ -│ ├── base/ -│ │ ├── infra/ # Core infrastructure (Cert-Manager, Namespaces, etc.) -│ │ └── matrix/ # HelmRepository definition for ESS OCI chart -│ ├── production/ # Main ESS deployment -│ │ ├── element-server-suite.yaml # HelmRelease (ESS chart v26.4.0) -│ │ ├── custom-configs/ # Overrides & custom configurations -│ │ │ ├── synapse-values.yaml # Synapse customizations (ConfigMap) -│ │ │ ├── element-values.yaml # ElementWeb customizations (ConfigMap) -│ │ │ └── mas-secret.yaml # MAS secrets (encrypted with SOPS) -│ │ ├── cert-issuer.yaml # Let's Encrypt ClusterIssuer -│ │ ├── apex-ingress.yaml # Apex-domain IngressRoutes (Element Web, /_scan, etc.) -│ │ ├── matrix-postgres-auth.yaml # PostgreSQL credentials -│ │ ├── coturn.yaml / coturn-secret.yaml / synapse-turn-secret.yaml -│ │ ├── turn-secret-rotation.yaml # Monthly CronJob, rotates coturn shared secret via PR -│ │ ├── draupnir.yaml / draupnir-pvc.yaml / draupnir-secret.yaml -│ │ ├── clamav.yaml / clamav-pvc.yaml / clamav_spam_checker.py # Synapse-side scan module -│ │ ├── clamav-http-scanner.py / -Dockerfile / .yaml # Client-side scan service -│ │ ├── synapse-backup.yaml / synapse-backup-secret.yaml -│ │ └── networkpolicy.yaml # Default-deny + explicit allow rules -│ ├── authentik/ # Identity Provider (separate namespace) -│ │ ├── authentik.yaml # HelmRelease -│ │ ├── authentik-blueprints.yaml # Flows/OIDC-provider as declarative code -│ │ ├── helm-repo.yaml # HelmRepository source -│ │ ├── ingress.yaml # Ingress route -│ │ ├── networkpolicy.yaml -│ │ └── authentik-secret.yaml # Secrets (admin password, OIDC client secret, etc.) -│ └── monitoring/ # Observability (Alloy, kube-state-metrics, node-exporter) -│ ├── alloy-config.yaml # Grafana Alloy configuration -│ └── kube-state-metrics.yaml # K8s metrics exporter -├── host-config/ # Host-level (non-GitOps) config, see below -│ └── maintenance-notify/ # systemd timer: pre-update mail/Matrix notifications (Issue #24) -├── .sops.yaml # SOPS encryption rules (age key definition) -├── scripts/ -│ ├── install-hooks.sh # Installs git hooks for ConfigMap auto-tracking -│ └── hooks/ # Git hooks (pre-commit, post-commit, etc.) -└── docs/ - ├── README.md # Main deployment guide - ├── TASKS.md # Task list & milestones (backlog itself lives in git.lab issues) - ├── install.md # Installation instructions - ├── ops-configmap-sync.md # ConfigMap syncing with git hooks - └── deployment-guides/ # Detailed guides for specific components (01-07) -``` - -## Host-Level (non-GitOps) Changes - -Almost everything in this repo is reconciled by Flux. `host-config/` is the deliberate -exception: it holds scripts/systemd units meant to run **on the bare Hetzner host itself** -(not as a Kubernetes pod), for things Flux structurally can't reach — e.g. host package -management. There is no SOPS-on-host or Ansible-equivalent mechanism yet; deployment to the -host is manual (`scp`/SSH), and instance-specific values live in a config file on the host -(`/etc//config`), not hardcoded in the versioned script, so the pattern is reusable -across forks/other communities running this same stack. See -`docs/deployment-guides/07-host-maintenance-notifications.md` for the first (and so far only) -example of this pattern. - -## Common Development Commands - -### Flux / GitOps Synchronization - -```bash -# Force immediate reconciliation (don't wait for 10-min auto-sync) -flux reconcile kustomization flux-system --with-source -flux reconcile kustomization production-apps --with-source - -# Check reconciliation status -flux get kustomizations -A -flux get helmreleases -A - -# View Flux logs -kubectl logs -n flux-system deployment/source-controller -f -kubectl logs -n flux-system deployment/helm-controller -f -``` - -### Kubernetes Cluster Status - -```bash -# Check pod health in Matrix namespace -kubectl get pods -n matrix -kubectl get pods -n authentik -kubectl get pods -n monitoring - -# Detailed pod inspection -kubectl describe pod -n matrix -kubectl logs -n matrix -f - -# Check all services and ingresses -kubectl get svc -n matrix -kubectl get ingress -n matrix -``` - -### Certificate Management (Let's Encrypt / Cert-Manager) - -```bash -# View certificate status -kubectl get certificate -n matrix -kubectl get certificaterequest -n matrix -kubectl get challenges -n matrix - -# Debug failed certificate issuance -kubectl describe challenge -n matrix -kubectl logs -n cert-manager deployment/cert-manager -f - -# Inspect the issued certificate -kubectl get secret -n matrix -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout -``` - -### SOPS Secret Editing - -SOPS transparently encrypts/decrypts secrets using the `age` key specified in `.sops.yaml`. The environment variable `SOPS_AGE_KEY_FILE` must point to your age private key. - -```bash -# Edit an encrypted secret (decrypted for editing, re-encrypted on save) -sops apps/production/custom-configs/mas-secret.yaml -sops apps/authentik/authentik-secret.yaml - -# Create a new secret file -sops -i --encrypted-regex '^(data|stringData)$' --input-type yaml --output-type yaml new-secret.yaml - -# Decrypt to view -sops -d apps/production/custom-configs/mas-secret.yaml -``` - -Ensure `~/.age/keys.txt` exists and contains your age private key. See `.devcontainer/devcontainer.json` for setup details. - -### Helm Chart Inspection - -```bash -# List installed charts -helm list -n matrix -helm list -n authentik - -# View rendered chart values -helm get values matrix-stack -n matrix -helm get manifest matrix-stack -n matrix | less -``` - -### Useful kubectl Shortcuts - -```bash -# Port-forward to access services locally -kubectl port-forward -n matrix svc/synapse 8008:8008 - -# Execute command inside pod (for debugging) -kubectl exec -it -n matrix -- bash - -# Stream logs from multiple pods -kubectl logs -n matrix -l app=synapse -f - -# Bootstrap a service/bot account via MAS (no registration_shared_secret in this stack) -kubectl exec -it -n matrix deploy/matrix-stack-matrix-authentication-service -- \ - mas-cli manage register-user --yes -kubectl exec -it -n matrix deploy/matrix-stack-matrix-authentication-service -- \ - mas-cli manage issue-compatibility-token -``` - -## Architecture & Key Concepts - -### FluxCD Reconciliation Flow - -1. **Flux watches** `clusters/matrix/` for a FluxRepository resource pointing to this Git repo -2. **Kustomization stages** pull in configurations in order: - - `flux-system` (FluxCD itself) - - `infra-apps` (Namespaces, RBAC, Cert-Manager, HelmRepository sources) - - `production-apps` (Main ESS deployment and related services) -3. **HelmReleases** specify which charts to install and what values to use -4. **ConfigMaps/Secrets** provide values from files in the repo (e.g., custom Synapse config) -5. **Flux auto-reconciles** every 10 minutes, or immediately if Git changes are detected - -### Element Server Suite (ESS) Chart Constraints - -The ESS Helm chart (v26.4.0) has strict validation and specific quirks: - -- **No `config:` blocks for core components** — use ConfigMap overrides instead -- **`serverName` must be at root level**, not nested under `synapse` -- **TLS in Ingress blocks is forbidden** — use `certManager: true` at root to auto-manage certificates -- **`camelCase` for component names**: `elementWeb`, `synapseAdmin`, `matrixAuthenticationService`, etc. -- **OCI HelmRepository only** — the chart is distributed via `oci://ghcr.io/element-hq/ess-helm`, not HTTP -- **Values must pass JSON schema validation** — invalid configs will cause reconciliation failures with cryptic schema errors - -### NetworkPolicy Convention - -Default-deny-with-explicit-allow across `matrix` and `authentik` namespaces -(`apps/production/networkpolicy.yaml`, `apps/authentik/networkpolicy.yaml`). Every new pod -needs its own explicit ingress-allow rule; NetworkPolicy matches on named **container ports**, -not Service ports — a frequent source of live incidents when a new component is added (wrong -port number/name silently blocks all traffic to it). - -### Known Issues & Workarounds - -**Issue: Let's Encrypt ACME Race Condition (Error 403 Order's status is processing)** -- Symptom: Certificate provisioning hangs when `elementWeb` and `wellKnownDelegation` are both enabled on the same domain -- Cause: Both request certificates for the same domain simultaneously; Let's Encrypt rejects concurrent requests -- Fix: Set `wellKnownDelegation: enabled: false` and serve `.well-known/matrix/server` via a separate Ingress route or static file - -**Issue: HelmChart not ready / stat no such file or directory** -- Cause: Attempting to use a GitRepository source for the ESS chart (it has sub-charts that don't render correctly) -- Fix: Use the OCI HelmRepository source (`oci://ghcr.io/element-hq/ess-helm`) instead - -**Issue: Certificate validation failures (No resources found)** -- Cause: Manual Kustomize patches conflict with the Helm chart's built-in certificate management -- Fix: Remove manual patches; rely on `certManager: true` at the root level of HelmRelease values - -**Issue: Synapse module can't use asyncio** -- Cause: Synapse runs on Twisted's reactor, not a running asyncio event loop — `asyncio.open_connection`/`asyncio.wait_for` inside a Synapse module (e.g. `clamav_spam_checker.py`) fail immediately with `RuntimeError: no running event loop`, and can silently trigger a fail-open path instead of an obvious crash -- Fix: use `twisted.internet.reactor`/`HostnameEndpoint`/`connectProtocol` + a custom `Protocol` subclass; Twisted `Deferred`s are natively awaitable from `async def` inside Synapse. Standalone processes outside Synapse (e.g. `clamav-http-scanner.py`) don't have this constraint and can use plain sockets/asyncio. - -### SOPS Encryption & Key Management - -- `.sops.yaml` defines encryption rules (currently using `age` keys) -- Secrets matching the regex in `.sops.yaml` are automatically encrypted when committed -- The age private key (`~/.age/keys.txt`) must be available in your environment for decryption -- In the cluster, Flux decrypts secrets "on the fly" using a secret stored in `flux-system` namespace - -To rotate SOPS keys: -```bash -# Regenerate and re-encrypt all secrets -sops updatekeys -y apps/ -``` - -## Development Workflow - -### Before Making Changes - -1. **Understand dependencies** — check `kustomization.yaml` files to see the order of resource creation -2. **Verify chart schema** — review ESS chart documentation for constraints on the version being used -3. **Test locally if possible** — use `kubectl` port-forwards to verify connectivity before pushing changes - -### Making Changes - -1. **Edit ConfigMap files directly** — for non-secret customizations (Synapse config, Element Web themes, etc.) - - Changes are auto-tracked by git hooks installed via `./scripts/install-hooks.sh` -2. **Edit secrets with SOPS** — `sops` transparently decrypts/re-encrypts on save -3. **Update HelmRelease values** — modify the `values` section in `element-server-suite.yaml` or reference ConfigMap sources - -### After Committing - -1. **Flux auto-detects changes** within ~1 minute (or manually trigger with `flux reconcile kustomization production-apps`) -2. **Monitor reconciliation** — watch pod logs and Flux status for errors -3. **Test functionality** — verify services are accessible and functioning as expected - -### Git Hooks - -After cloning, run: -```bash -./scripts/install-hooks.sh -``` - -This installs hooks that automatically commit ConfigMap changes to `.gitignore`-like tracking. See `docs/ops-configmap-sync.md` for details. - -## Environment Setup - -### Local Machine Prerequisites - -- `kubectl` — cluster communication -- `flux` — GitOps CLI -- `helm` — chart inspection & debugging -- `sops` & `age` — secret management -- `git` — version control -- age key file at `~/.age/keys.txt` (request from team) -- kubeconfig at `~/.kube/config` (request from team) - -### DevContainer (Recommended) - -The `.devcontainer/` configuration provides a pre-configured environment: - -```bash -# In VS Code: "Reopen in Container" -# Or manually: -docker build -t ess-devcontainer .devcontainer -docker run -it --rm \ - -v ~/.kube:/home/vscode/.kube \ - -v ~/.age:/home/vscode/.age \ - -v ~/.ssh:/home/vscode/.ssh \ - -v /var/run/docker.sock:/var/run/docker.sock \ - ess-devcontainer -``` - -DevContainer includes: -- All required CLI tools (kubectl, flux, helm, sops, age, git, docker) -- VS Code extensions for YAML, Kubernetes, Helm -- Proper environment variables (`KUBECONFIG`, `SOPS_AGE_KEY_FILE`) -- Git hooks pre-installed - -## Troubleshooting Checklist - -- **Pod not starting?** → `kubectl describe pod -n matrix` (check events) -- **Image pull failures?** → Check HelmRelease status: `kubectl get helmrelease -n matrix` -- **Secret not found?** → Verify SOPS decryption: `sops -d ` (must output valid YAML) -- **Certificate stuck?** → `kubectl describe certificate -n matrix` (check for ACME errors) -- **Config validation error?** → Inspect HelmRelease status: `kubectl describe helmrelease -n matrix` (JSON schema error message) -- **Cluster unreachable?** → Verify kubeconfig: `kubectl get nodes` (must connect to K3s) -- **NetworkPolicy blocking a new pod?** → Check it matches on container port name, not Service port - -## Resources & References - -- **README.md** — High-level overview and architecture -- **docs/TASKS.md** — Task backlog, milestones, and priority list (open backlog lives in [git.lab issues](https://git.lab/axion1337.chat/axion1337.chat-gitops/-/issues); the closed Gitea issues only point there) -- **docs/deployment-guides/** — Detailed setup guides for specific components (01-07) -- **docs/ops-configmap-sync.md** — Git hook configuration and auto-sync behavior -- **ESS Chart Docs** — `https://github.com/element-hq/ess-helm` (official Helm chart repository) -- **FluxCD Docs** — `https://fluxcd.io/docs/` (GitOps reconciliation & Kustomization) -- **Matrix Spec** — `https://spec.matrix.org/` (Matrix protocol specification) - ---- -name: karpathy-guidelines -description: Behavioral guidelines to reduce common LLM coding mistakes. Use when writing, reviewing, or refactoring code to avoid overcomplication, make surgical changes, surface assumptions, and define verifiable success criteria. -license: MIT ---- - -# Karpathy Guidelines - -Behavioral guidelines to reduce common LLM coding mistakes, derived from [Andrej Karpathy's observations](https://x.com/karpathy/status/2015883857489522876) on LLM coding pitfalls. - -**Tradeoff:** These guidelines bias toward caution over speed. For trivial tasks, use judgment. - -## 1. Think Before Coding - -**Don't assume. Don't hide confusion. Surface tradeoffs.** - -Before implementing: -- State your assumptions explicitly. If uncertain, ask. -- If multiple interpretations exist, present them - don't pick silently. -- If a simpler approach exists, say so. Push back when warranted. -- If something is unclear, stop. Name what's confusing. Ask. - -## 2. Simplicity First - -**Minimum code that solves the problem. Nothing speculative.** - -- No features beyond what was asked. -- No abstractions for single-use code. -- No "flexibility" or "configurability" that wasn't requested. -- No error handling for impossible scenarios. -- If you write 200 lines and it could be 50, rewrite it. - -Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify. - -## 3. Surgical Changes - -**Touch only what you must. Clean up only your own mess.** - -When editing existing code: -- Don't "improve" adjacent code, comments, or formatting. -- Don't refactor things that aren't broken. -- Match existing style, even if you'd do it differently. -- If you notice unrelated dead code, mention it - don't delete it. - -When your changes create orphans: -- Remove imports/variables/functions that YOUR changes made unused. -- Don't remove pre-existing dead code unless asked. - -The test: Every changed line should trace directly to the user's request. - -## 4. Goal-Driven Execution - -**Define success criteria. Loop until verified.** - -Transform tasks into verifiable goals: -- "Add validation" → "Write tests for invalid inputs, then make them pass" -- "Fix the bug" → "Write a test that reproduces it, then make it pass" -- "Refactor X" → "Ensure tests pass before and after" - -For multi-step tasks, state a brief plan: -``` -1. [Step] → verify: [check] -2. [Step] → verify: [check] -3. [Step] → verify: [check] -``` - -Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification. +Read AGENTS.md — the canonical instruction file for this repository. All rules live there.