# CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. > **Cross-project working conventions are canonical in the `management` repo's > [`CLAUDE.md`](https://git.lab/axion1337.chat/management/-/blob/main/CLAUDE.md)** > (readable from anywhere via the Gitea mirror > `https://rohana.axion1337.de/sorb/management`): repo topology & mirror rules, > the Kanban framework (status labels, WIP limit 2, ADR duty — including for > permanent rule exceptions), deploy-handover/AAR procedures, secrets handling, > and the Karpathy guidelines. Every agent session must follow it; this file > covers only what is specific to this repository. ## Overview This is a **GitOps-based Kubernetes deployment** of **Element Server Suite (ESS Community v26.4.0)**, a complete Matrix homeserver stack. The repository contains Infrastructure-as-Code using **FluxCD** for GitOps synchronization, with encryption (SOPS/age), service mesh (Traefik), certificate management (Cert-Manager), and auxiliary services like Authentik, TURN/coturn, Draupnir (moderation), ClamAV (content scanning), and Grafana monitoring. **Key Stack Components:** - **K3s**: Lightweight Kubernetes distribution running on Hetzner Cloud - **FluxCD**: GitOps controller that watches this repository and auto-syncs changes - **ESS (Matrix Umbrella Chart v26.4.0)**: Synapse, MAS (Matrix Authentication Service), ElementWeb, MatrixRTC - **Authentik**: OIDC-based identity provider for centralized authentication, deployed both via HelmRelease and declarative Blueprints (`apps/authentik/authentik-blueprints.yaml`) for flows/OIDC-provider config that would otherwise only exist as manual admin-UI clicks - **Traefik**: Ingress controller (built into K3s) for routing HTTP/HTTPS traffic - **Cert-Manager**: Automatic TLS certificate provisioning from Let's Encrypt - **SOPS + age**: Transparent encryption/decryption of secrets in Git - **Monitoring**: Grafana Alloy (agent), Prometheus (metrics), Loki (logs) - **coturn**: TURN/STUN server for WebRTC audio/video calls, with monthly automated shared-secret rotation via CronJob + PR workflow - **Draupnir**: Matrix moderation bot (community successor to Mjolnir), ban lists/policy rooms - **ClamAV**: Content scanning — a Synapse module for unencrypted-room uploads, plus a standalone `clamav-http-scanner` service that a patched Element Web client (ThreadNet-Web) calls both on send and on receive, extending coverage to encrypted rooms/DMs - **NetworkPolicies**: default-deny-with-explicit-allow across `matrix` and `authentik` namespaces - **`host-config/`**: the one part of this repo that is deliberately **not** managed by Flux/GitOps — see "Host-Level (non-GitOps) Changes" below ## Repo Topology (since 2026-07-31) Canonical repo is **`git.lab/axion1337.chat/axion1337.chat-gitops`** (homelab GitLab, resolvable only inside the lab) — all pushes go there; a push-mirror updates the Gitea copy on `rohana.axion1337.de`, which remains the **Flux source** (the cluster pulls from Gitea; the mirror delivers). **Never push directly to Gitea** for this repo — the mirror force-overwrites divergent state. The same rule applies to ThreadNet-Web, threadnet-call, thread-net-git, threadnet-operating and (since 2026-08-01) `management` (the former `Backlogs` repo, renamed in the PM-framework restructuring — ADRs/vision/roadmap live there). Since 2026-08-02 the wiki lives on git.lab too, so **no repo is authored on Gitea any more**; the one process that still *writes* there is the TURN rotation (see below). **Why Gitea is the Flux source, and why that is not a leftover.** git.lab holds the blueprints; Gitea holds a copy the cluster can reach without the lab being up. That separation is deliberate: the Hetzner cluster must be buildable and re-deployable when the homelab is offline, on holiday, or mid-rebuild — it therefore must not depend on a host that only answers inside the lab. **Do not "fix" the Flux source to point at git.lab**: it would look tidier and would couple production availability to the lab, which is exactly what this split avoids. **Issues live on git.lab** (migrated 2026-08-01, gitops#48): the old Gitea issues are closed with a pointer to their GitLab counterpart. ⚠️ gitops issue numbers **shifted** in the migration (Gitea had gaps from PRs; e.g. Gitea#48 → GitLab#46) — old references "gitops#N" in commits/docs mean the **Gitea** number; the authoritative mapping is the migration footer in each GitLab issue. Releases stay on Gitea (public download path). **No exceptions left**: deploy-handover issues used to stay on the Gitea `sorb/management` tracker because CFGMON had no route to git.lab. The site-to-site VPN (ADR-0004) removed that reason, and on 2026-08-02 they moved (LABNET-03) — the Gitea tracker is empty and the form now lives as a GitLab issue template in the management repo. Issues live on git.lab, without exception. **Wiki** (migrated 2026-08-02): the 15 pages now live in this project's GitLab wiki (`…-gitops.wiki.git`, *Wiki* tab). ⚠️ The **`wiki` branch in this repo is NOT the wiki** — it is a stale May snapshot of `docs/`; don't edit or trust it. All doc sources (platform wiki, `homelab/docs`, `management`) are also served side by side as one Docusaurus site at **wiki.lab**, configured in `git.lab/homelab/wiki` — content is pulled at build time, so edits always belong in the source repo. **The one write that still lands on Gitea**: the monthly TURN-rotation CronJob runs in the cluster (no route to git.lab) and pushes its rotation branch to Gitea. Never merge that PR on Gitea — the rotation has to travel back through git.lab, otherwise the next mirror push overwrites it and Flux silently re-applies the *old* shared secret. **This is automated — do not do it by hand.** The scheduled job `canonize_rotation` in `.gitlab-ci.yml` runs daily on git.lab, picks up any `turn-secret-rotation-*` branch from Gitea that is not yet in `main`, merges it, and pushes to git.lab; the mirror carries it back and Gitea auto-closes the PR. Once merged, the branch exists only on Gitea, so the next mirror run deletes it — no cleanup needed. If the job fails (merge conflict, or the SOPS check finds an unencrypted secret file), it fails **loudly and changes nothing** — the pipeline stays red until someone looks. That red pipeline is the alarm; there is no separate reminder. Manual fallback, should it ever be needed: fetch the branch, merge into main locally, push to git.lab (worked example: 2026-08-01, commit `640c934`). If the mirror lags, force it via the GitLab API `POST /projects//remote_mirrors//sync`. ## Repository Structure ``` gitops/ ├── .gitlab-ci.yml # Lab-CI verify job (YAML validation + SOPS-encryption check) ├── clusters/matrix/ # Flux GitRepository definition; entry point for reconciliation ├── apps/ │ ├── base/ │ │ ├── infra/ # Core infrastructure (Cert-Manager, Namespaces, etc.) │ │ └── matrix/ # HelmRepository definition for ESS OCI chart │ ├── production/ # Main ESS deployment │ │ ├── element-server-suite.yaml # HelmRelease (ESS chart v26.4.0) │ │ ├── custom-configs/ # Overrides & custom configurations │ │ │ ├── synapse-values.yaml # Synapse customizations (ConfigMap) │ │ │ ├── element-values.yaml # ElementWeb customizations (ConfigMap) │ │ │ └── mas-secret.yaml # MAS secrets (encrypted with SOPS) │ │ ├── cert-issuer.yaml # Let's Encrypt ClusterIssuer │ │ ├── apex-ingress.yaml # Apex-domain IngressRoutes (Element Web, /_scan, etc.) │ │ ├── matrix-postgres-auth.yaml # PostgreSQL credentials │ │ ├── coturn.yaml / coturn-secret.yaml / synapse-turn-secret.yaml │ │ ├── turn-secret-rotation.yaml # Monthly CronJob, rotates coturn shared secret via PR │ │ ├── draupnir.yaml / draupnir-pvc.yaml / draupnir-secret.yaml │ │ ├── clamav.yaml / clamav-pvc.yaml / clamav_spam_checker.py # Synapse-side scan module │ │ ├── clamav-http-scanner.py / -Dockerfile / .yaml # Client-side scan service │ │ ├── synapse-backup.yaml / synapse-backup-secret.yaml │ │ └── networkpolicy.yaml # Default-deny + explicit allow rules │ ├── authentik/ # Identity Provider (separate namespace) │ │ ├── authentik.yaml # HelmRelease │ │ ├── authentik-blueprints.yaml # Flows/OIDC-provider as declarative code │ │ ├── helm-repo.yaml # HelmRepository source │ │ ├── ingress.yaml # Ingress route │ │ ├── networkpolicy.yaml │ │ └── authentik-secret.yaml # Secrets (admin password, OIDC client secret, etc.) │ └── monitoring/ # Observability (Alloy, kube-state-metrics, node-exporter) │ ├── alloy-config.yaml # Grafana Alloy configuration │ └── kube-state-metrics.yaml # K8s metrics exporter ├── host-config/ # Host-level (non-GitOps) config, see below │ └── maintenance-notify/ # systemd timer: pre-update mail/Matrix notifications (Issue #24) ├── .sops.yaml # SOPS encryption rules (age key definition) ├── scripts/ │ ├── install-hooks.sh # Installs git hooks for ConfigMap auto-tracking │ └── hooks/ # Git hooks (pre-commit, post-commit, etc.) └── docs/ ├── README.md # Main deployment guide ├── TASKS.md # Task list & milestones (backlog itself lives in git.lab issues) ├── install.md # Installation instructions ├── ops-configmap-sync.md # ConfigMap syncing with git hooks └── deployment-guides/ # Detailed guides for specific components (01-07) ``` ## Host-Level (non-GitOps) Changes Almost everything in this repo is reconciled by Flux. `host-config/` is the deliberate exception: it holds scripts/systemd units meant to run **on the bare Hetzner host itself** (not as a Kubernetes pod), for things Flux structurally can't reach — e.g. host package management. There is no SOPS-on-host or Ansible-equivalent mechanism yet; deployment to the host is manual (`scp`/SSH), and instance-specific values live in a config file on the host (`/etc//config`), not hardcoded in the versioned script, so the pattern is reusable across forks/other communities running this same stack. See `docs/deployment-guides/07-host-maintenance-notifications.md` for the first (and so far only) example of this pattern. ## Common Development Commands ### Flux / GitOps Synchronization ```bash # Force immediate reconciliation (don't wait for 10-min auto-sync) flux reconcile kustomization flux-system --with-source flux reconcile kustomization production-apps --with-source # Check reconciliation status flux get kustomizations -A flux get helmreleases -A # View Flux logs kubectl logs -n flux-system deployment/source-controller -f kubectl logs -n flux-system deployment/helm-controller -f ``` ### Kubernetes Cluster Status ```bash # Check pod health in Matrix namespace kubectl get pods -n matrix kubectl get pods -n authentik kubectl get pods -n monitoring # Detailed pod inspection kubectl describe pod -n matrix kubectl logs -n matrix -f # Check all services and ingresses kubectl get svc -n matrix kubectl get ingress -n matrix ``` ### Certificate Management (Let's Encrypt / Cert-Manager) ```bash # View certificate status kubectl get certificate -n matrix kubectl get certificaterequest -n matrix kubectl get challenges -n matrix # Debug failed certificate issuance kubectl describe challenge -n matrix kubectl logs -n cert-manager deployment/cert-manager -f # Inspect the issued certificate kubectl get secret -n matrix -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout ``` ### SOPS Secret Editing SOPS transparently encrypts/decrypts secrets using the `age` key specified in `.sops.yaml`. The environment variable `SOPS_AGE_KEY_FILE` must point to your age private key. ```bash # Edit an encrypted secret (decrypted for editing, re-encrypted on save) sops apps/production/custom-configs/mas-secret.yaml sops apps/authentik/authentik-secret.yaml # Create a new secret file sops -i --encrypted-regex '^(data|stringData)$' --input-type yaml --output-type yaml new-secret.yaml # Decrypt to view sops -d apps/production/custom-configs/mas-secret.yaml ``` Ensure `~/.age/keys.txt` exists and contains your age private key. See `.devcontainer/devcontainer.json` for setup details. ### Helm Chart Inspection ```bash # List installed charts helm list -n matrix helm list -n authentik # View rendered chart values helm get values matrix-stack -n matrix helm get manifest matrix-stack -n matrix | less ``` ### Useful kubectl Shortcuts ```bash # Port-forward to access services locally kubectl port-forward -n matrix svc/synapse 8008:8008 # Execute command inside pod (for debugging) kubectl exec -it -n matrix -- bash # Stream logs from multiple pods kubectl logs -n matrix -l app=synapse -f # Bootstrap a service/bot account via MAS (no registration_shared_secret in this stack) kubectl exec -it -n matrix deploy/matrix-stack-matrix-authentication-service -- \ mas-cli manage register-user --yes kubectl exec -it -n matrix deploy/matrix-stack-matrix-authentication-service -- \ mas-cli manage issue-compatibility-token ``` ## Architecture & Key Concepts ### FluxCD Reconciliation Flow 1. **Flux watches** `clusters/matrix/` for a FluxRepository resource pointing to this Git repo 2. **Kustomization stages** pull in configurations in order: - `flux-system` (FluxCD itself) - `infra-apps` (Namespaces, RBAC, Cert-Manager, HelmRepository sources) - `production-apps` (Main ESS deployment and related services) 3. **HelmReleases** specify which charts to install and what values to use 4. **ConfigMaps/Secrets** provide values from files in the repo (e.g., custom Synapse config) 5. **Flux auto-reconciles** every 10 minutes, or immediately if Git changes are detected ### Element Server Suite (ESS) Chart Constraints The ESS Helm chart (v26.4.0) has strict validation and specific quirks: - **No `config:` blocks for core components** — use ConfigMap overrides instead - **`serverName` must be at root level**, not nested under `synapse` - **TLS in Ingress blocks is forbidden** — use `certManager: true` at root to auto-manage certificates - **`camelCase` for component names**: `elementWeb`, `synapseAdmin`, `matrixAuthenticationService`, etc. - **OCI HelmRepository only** — the chart is distributed via `oci://ghcr.io/element-hq/ess-helm`, not HTTP - **Values must pass JSON schema validation** — invalid configs will cause reconciliation failures with cryptic schema errors ### NetworkPolicy Convention Default-deny-with-explicit-allow across `matrix` and `authentik` namespaces (`apps/production/networkpolicy.yaml`, `apps/authentik/networkpolicy.yaml`). Every new pod needs its own explicit ingress-allow rule; NetworkPolicy matches on named **container ports**, not Service ports — a frequent source of live incidents when a new component is added (wrong port number/name silently blocks all traffic to it). ### Known Issues & Workarounds **Issue: Let's Encrypt ACME Race Condition (Error 403 Order's status is processing)** - Symptom: Certificate provisioning hangs when `elementWeb` and `wellKnownDelegation` are both enabled on the same domain - Cause: Both request certificates for the same domain simultaneously; Let's Encrypt rejects concurrent requests - Fix: Set `wellKnownDelegation: enabled: false` and serve `.well-known/matrix/server` via a separate Ingress route or static file **Issue: HelmChart not ready / stat no such file or directory** - Cause: Attempting to use a GitRepository source for the ESS chart (it has sub-charts that don't render correctly) - Fix: Use the OCI HelmRepository source (`oci://ghcr.io/element-hq/ess-helm`) instead **Issue: Certificate validation failures (No resources found)** - Cause: Manual Kustomize patches conflict with the Helm chart's built-in certificate management - Fix: Remove manual patches; rely on `certManager: true` at the root level of HelmRelease values **Issue: Synapse module can't use asyncio** - Cause: Synapse runs on Twisted's reactor, not a running asyncio event loop — `asyncio.open_connection`/`asyncio.wait_for` inside a Synapse module (e.g. `clamav_spam_checker.py`) fail immediately with `RuntimeError: no running event loop`, and can silently trigger a fail-open path instead of an obvious crash - Fix: use `twisted.internet.reactor`/`HostnameEndpoint`/`connectProtocol` + a custom `Protocol` subclass; Twisted `Deferred`s are natively awaitable from `async def` inside Synapse. Standalone processes outside Synapse (e.g. `clamav-http-scanner.py`) don't have this constraint and can use plain sockets/asyncio. ### SOPS Encryption & Key Management - `.sops.yaml` defines encryption rules (currently using `age` keys) - Secrets matching the regex in `.sops.yaml` are automatically encrypted when committed - The age private key (`~/.age/keys.txt`) must be available in your environment for decryption - In the cluster, Flux decrypts secrets "on the fly" using a secret stored in `flux-system` namespace To rotate SOPS keys: ```bash # Regenerate and re-encrypt all secrets sops updatekeys -y apps/ ``` ## Development Workflow ### Before Making Changes 1. **Understand dependencies** — check `kustomization.yaml` files to see the order of resource creation 2. **Verify chart schema** — review ESS chart documentation for constraints on the version being used 3. **Test locally if possible** — use `kubectl` port-forwards to verify connectivity before pushing changes ### Making Changes 1. **Edit ConfigMap files directly** — for non-secret customizations (Synapse config, Element Web themes, etc.) - Changes are auto-tracked by git hooks installed via `./scripts/install-hooks.sh` 2. **Edit secrets with SOPS** — `sops` transparently decrypts/re-encrypts on save 3. **Update HelmRelease values** — modify the `values` section in `element-server-suite.yaml` or reference ConfigMap sources ### After Committing 1. **Flux auto-detects changes** within ~1 minute (or manually trigger with `flux reconcile kustomization production-apps`) 2. **Monitor reconciliation** — watch pod logs and Flux status for errors 3. **Test functionality** — verify services are accessible and functioning as expected ### Git Hooks After cloning, run: ```bash ./scripts/install-hooks.sh ``` This installs hooks that automatically commit ConfigMap changes to `.gitignore`-like tracking. See `docs/ops-configmap-sync.md` for details. ## Environment Setup ### Local Machine Prerequisites - `kubectl` — cluster communication - `flux` — GitOps CLI - `helm` — chart inspection & debugging - `sops` & `age` — secret management - `git` — version control - age key file at `~/.age/keys.txt` (request from team) - kubeconfig at `~/.kube/config` (request from team) ### DevContainer (Recommended) The `.devcontainer/` configuration provides a pre-configured environment: ```bash # In VS Code: "Reopen in Container" # Or manually: docker build -t ess-devcontainer .devcontainer docker run -it --rm \ -v ~/.kube:/home/vscode/.kube \ -v ~/.age:/home/vscode/.age \ -v ~/.ssh:/home/vscode/.ssh \ -v /var/run/docker.sock:/var/run/docker.sock \ ess-devcontainer ``` DevContainer includes: - All required CLI tools (kubectl, flux, helm, sops, age, git, docker) - VS Code extensions for YAML, Kubernetes, Helm - Proper environment variables (`KUBECONFIG`, `SOPS_AGE_KEY_FILE`) - Git hooks pre-installed ## Troubleshooting Checklist - **Pod not starting?** → `kubectl describe pod -n matrix` (check events) - **Image pull failures?** → Check HelmRelease status: `kubectl get helmrelease -n matrix` - **Secret not found?** → Verify SOPS decryption: `sops -d ` (must output valid YAML) - **Certificate stuck?** → `kubectl describe certificate -n matrix` (check for ACME errors) - **Config validation error?** → Inspect HelmRelease status: `kubectl describe helmrelease -n matrix` (JSON schema error message) - **Cluster unreachable?** → Verify kubeconfig: `kubectl get nodes` (must connect to K3s) - **NetworkPolicy blocking a new pod?** → Check it matches on container port name, not Service port ## Resources & References - **README.md** — High-level overview and architecture - **docs/TASKS.md** — Task backlog, milestones, and priority list (open backlog lives in [git.lab issues](https://git.lab/axion1337.chat/axion1337.chat-gitops/-/issues); the closed Gitea issues only point there) - **docs/deployment-guides/** — Detailed setup guides for specific components (01-07) - **docs/ops-configmap-sync.md** — Git hook configuration and auto-sync behavior - **ESS Chart Docs** — `https://github.com/element-hq/ess-helm` (official Helm chart repository) - **FluxCD Docs** — `https://fluxcd.io/docs/` (GitOps reconciliation & Kustomization) - **Matrix Spec** — `https://spec.matrix.org/` (Matrix protocol specification) --- name: karpathy-guidelines description: Behavioral guidelines to reduce common LLM coding mistakes. Use when writing, reviewing, or refactoring code to avoid overcomplication, make surgical changes, surface assumptions, and define verifiable success criteria. license: MIT --- # Karpathy Guidelines Behavioral guidelines to reduce common LLM coding mistakes, derived from [Andrej Karpathy's observations](https://x.com/karpathy/status/2015883857489522876) on LLM coding pitfalls. **Tradeoff:** These guidelines bias toward caution over speed. For trivial tasks, use judgment. ## 1. Think Before Coding **Don't assume. Don't hide confusion. Surface tradeoffs.** Before implementing: - State your assumptions explicitly. If uncertain, ask. - If multiple interpretations exist, present them - don't pick silently. - If a simpler approach exists, say so. Push back when warranted. - If something is unclear, stop. Name what's confusing. Ask. ## 2. Simplicity First **Minimum code that solves the problem. Nothing speculative.** - No features beyond what was asked. - No abstractions for single-use code. - No "flexibility" or "configurability" that wasn't requested. - No error handling for impossible scenarios. - If you write 200 lines and it could be 50, rewrite it. Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify. ## 3. Surgical Changes **Touch only what you must. Clean up only your own mess.** When editing existing code: - Don't "improve" adjacent code, comments, or formatting. - Don't refactor things that aren't broken. - Match existing style, even if you'd do it differently. - If you notice unrelated dead code, mention it - don't delete it. When your changes create orphans: - Remove imports/variables/functions that YOUR changes made unused. - Don't remove pre-existing dead code unless asked. The test: Every changed line should trace directly to the user's request. ## 4. Goal-Driven Execution **Define success criteria. Loop until verified.** Transform tasks into verifiable goals: - "Add validation" → "Write tests for invalid inputs, then make them pass" - "Fix the bug" → "Write a test that reproduces it, then make it pass" - "Refactor X" → "Ensure tests pass before and after" For multi-step tasks, state a brief plan: ``` 1. [Step] → verify: [check] 2. [Step] → verify: [check] 3. [Step] → verify: [check] ``` Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification.