sorb/axion1337.chat-gitops
1
0
Fork 0
Code Issues Pull Requests Packages Projects 1 Releases 24 Wiki Activity

docs: split CLAUDE.md into a pointer plus AGENTS.md

Browse Source 
Follows the management repo's shape (ADR-0013): CLAUDE.md is the one-line pointer
the group check looks for, AGENTS.md carries the project specifics unchanged. The
group-rules reference now points at management's AGENTS.md rather than its
CLAUDE.md, which had itself become a pointer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Thore Cimbal
2026-08-15 12:00:00 +00:00
co-authored by Claude Opus 4.8
parent 60aaf0e068
commit 23533c7899
2 changed files with 465 additions and 466 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+1 -466
View File
@@ -1,466 +1 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
> **Cross-project working conventions are canonical in the `management` repo's
> [`CLAUDE.md`](https://git.lab/axion1337.chat/management/-/blob/main/CLAUDE.md)**
> (readable from anywhere via the Gitea mirror
> `https://rohana.axion1337.de/sorb/management`): repo topology & mirror rules,
> the Kanban framework (status labels, WIP limit 2, ADR duty — including for
> permanent rule exceptions), deploy-handover/AAR procedures, secrets handling,
> and the Karpathy guidelines. Every agent session must follow it; this file
> covers only what is specific to this repository.
## Overview
This is a **GitOps-based Kubernetes deployment** of **Element Server Suite (ESS Community v26.4.0)**, a complete Matrix homeserver stack. The repository contains Infrastructure-as-Code using **FluxCD** for GitOps synchronization, with encryption (SOPS/age), service mesh (Traefik), certificate management (Cert-Manager), and auxiliary services like Authentik, TURN/coturn, Draupnir (moderation), ClamAV (content scanning), and Grafana monitoring.
**Key Stack Components:**
- **K3s**: Lightweight Kubernetes distribution running on Hetzner Cloud
- **FluxCD**: GitOps controller that watches this repository and auto-syncs changes
- **ESS (Matrix Umbrella Chart v26.4.0)**: Synapse, MAS (Matrix Authentication Service), ElementWeb, MatrixRTC
- **Authentik**: OIDC-based identity provider for centralized authentication, deployed both via HelmRelease and declarative Blueprints (`apps/authentik/authentik-blueprints.yaml`) for flows/OIDC-provider config that would otherwise only exist as manual admin-UI clicks
- **Traefik**: Ingress controller (built into K3s) for routing HTTP/HTTPS traffic
- **Cert-Manager**: Automatic TLS certificate provisioning from Let's Encrypt
- **SOPS + age**: Transparent encryption/decryption of secrets in Git
- **Monitoring**: Grafana Alloy (agent), Prometheus (metrics), Loki (logs)
- **coturn**: TURN/STUN server for WebRTC audio/video calls, with monthly automated shared-secret rotation via CronJob + PR workflow
- **Draupnir**: Matrix moderation bot (community successor to Mjolnir), ban lists/policy rooms
- **ClamAV**: Content scanning — a Synapse module for unencrypted-room uploads, plus a standalone `clamav-http-scanner` service that a patched Element Web client (ThreadNet-Web) calls both on send and on receive, extending coverage to encrypted rooms/DMs
- **NetworkPolicies**: default-deny-with-explicit-allow across `matrix` and `authentik` namespaces
- **`host-config/`**: the one part of this repo that is deliberately **not** managed by Flux/GitOps — see "Host-Level (non-GitOps) Changes" below
## Repo Topology (since 2026-07-31)
Canonical repo is **`git.lab/axion1337.chat/axion1337.chat-gitops`** (homelab GitLab,
resolvable only inside the lab) — all pushes go there; a push-mirror updates the Gitea
copy on `rohana.axion1337.de`, which remains the **Flux source** (the cluster pulls from
Gitea; the mirror delivers). **Never push directly to Gitea** for this repo — the mirror
force-overwrites divergent state. The same rule applies to ThreadNet-Web, threadnet-call,
thread-net-git, threadnet-operating and (since 2026-08-01) `management` (the former
`Backlogs` repo, renamed in the PM-framework restructuring — ADRs/vision/roadmap live
there). Since 2026-08-02 the wiki lives on git.lab too, so **no repo is authored on
Gitea any more**; the one process that still *writes* there is the TURN rotation
(see below).
**Why Gitea is the Flux source, and why that is not a leftover.** git.lab holds the
blueprints; Gitea holds a copy the cluster can reach without the lab being up. That
separation is deliberate: the Hetzner cluster must be buildable and re-deployable when
the homelab is offline, on holiday, or mid-rebuild — it therefore must not depend on a
host that only answers inside the lab. **Do not "fix" the Flux source to point at
git.lab**: it would look tidier and would couple production availability to the lab,
which is exactly what this split avoids.
**Issues live on git.lab** (migrated 2026-08-01, gitops#48): the old Gitea issues are
closed with a pointer to their GitLab counterpart. ⚠️ gitops issue numbers **shifted**
in the migration (Gitea had gaps from PRs; e.g. Gitea#48 → GitLab#46) — old references
"gitops#N" in commits/docs mean the **Gitea** number; the authoritative mapping is the
migration footer in each GitLab issue. Releases stay on Gitea (public download path).
**No exceptions left**: deploy-handover issues used to stay on the Gitea
`sorb/management` tracker because CFGMON had no route to git.lab. The site-to-site VPN
(ADR-0004) removed that reason, and on 2026-08-02 they moved (LABNET-03) — the Gitea
tracker is empty and the form now lives as a GitLab issue template in the management
repo. Issues live on git.lab, without exception.
**Wiki** (migrated 2026-08-02): the 15 pages now live in this project's GitLab wiki
(`…-gitops.wiki.git`, *Wiki* tab). ⚠️ The **`wiki` branch in this repo is NOT the wiki** —
it is a stale May snapshot of `docs/`; don't edit or trust it. All doc sources
(platform wiki, `homelab/docs`, `management`) are also served side by side as one
Docusaurus site at **wiki.lab**, configured in `git.lab/homelab/wiki` — content is
pulled at build time, so edits always belong in the source repo.
**The one write that still lands on Gitea**: the monthly TURN-rotation CronJob runs in
the cluster (no route to git.lab) and pushes its rotation branch to Gitea. Never merge
that PR on Gitea — the rotation has to travel back through git.lab, otherwise the next
mirror push overwrites it and Flux silently re-applies the *old* shared secret.
**This is automated — do not do it by hand.** The scheduled job `canonize_rotation` in
`.gitlab-ci.yml` runs daily on git.lab, picks up any `turn-secret-rotation-*` branch
from Gitea that is not yet in `main`, merges it, and pushes to git.lab; the mirror
carries it back and Gitea auto-closes the PR. Once merged, the branch exists only on
Gitea, so the next mirror run deletes it — no cleanup needed.
If the job fails (merge conflict, or the SOPS check finds an unencrypted secret file),
it fails **loudly and changes nothing** — the pipeline stays red until someone looks.
That red pipeline is the alarm; there is no separate reminder. Manual fallback, should
it ever be needed: fetch the branch, merge into main locally, push to git.lab (worked
example: 2026-08-01, commit `717638d`). If the mirror lags, force it via the GitLab API
`POST /projects/<id>/remote_mirrors/<mirror_id>/sync`.
## Repository Structure
```
gitops/
├── .gitlab-ci.yml # Lab-CI verify job (YAML validation + SOPS-encryption check)
├── clusters/matrix/ # Flux GitRepository definition; entry point for reconciliation
├── apps/
│ ├── base/
│ │ ├── infra/ # Core infrastructure (Cert-Manager, Namespaces, etc.)
│ │ └── matrix/ # HelmRepository definition for ESS OCI chart
│ ├── production/ # Main ESS deployment
│ │ ├── element-server-suite.yaml # HelmRelease (ESS chart v26.4.0)
│ │ ├── custom-configs/ # Overrides & custom configurations
│ │ │ ├── synapse-values.yaml # Synapse customizations (ConfigMap)
│ │ │ ├── element-values.yaml # ElementWeb customizations (ConfigMap)
│ │ │ └── mas-secret.yaml # MAS secrets (encrypted with SOPS)
│ │ ├── cert-issuer.yaml # Let's Encrypt ClusterIssuer
│ │ ├── apex-ingress.yaml # Apex-domain IngressRoutes (Element Web, /_scan, etc.)
│ │ ├── matrix-postgres-auth.yaml # PostgreSQL credentials
│ │ ├── coturn.yaml / coturn-secret.yaml / synapse-turn-secret.yaml
│ │ ├── turn-secret-rotation.yaml # Monthly CronJob, rotates coturn shared secret via PR
│ │ ├── draupnir.yaml / draupnir-pvc.yaml / draupnir-secret.yaml
│ │ ├── clamav.yaml / clamav-pvc.yaml / clamav_spam_checker.py # Synapse-side scan module
│ │ ├── clamav-http-scanner.py / -Dockerfile / .yaml # Client-side scan service
│ │ ├── synapse-backup.yaml / synapse-backup-secret.yaml
│ │ └── networkpolicy.yaml # Default-deny + explicit allow rules
│ ├── authentik/ # Identity Provider (separate namespace)
│ │ ├── authentik.yaml # HelmRelease
│ │ ├── authentik-blueprints.yaml # Flows/OIDC-provider as declarative code
│ │ ├── helm-repo.yaml # HelmRepository source
│ │ ├── ingress.yaml # Ingress route
│ │ ├── networkpolicy.yaml
│ │ └── authentik-secret.yaml # Secrets (admin password, OIDC client secret, etc.)
│ └── monitoring/ # Observability (Alloy, kube-state-metrics, node-exporter)
│ ├── alloy-config.yaml # Grafana Alloy configuration
│ └── kube-state-metrics.yaml # K8s metrics exporter
├── host-config/ # Host-level (non-GitOps) config, see below
│ └── maintenance-notify/ # systemd timer: pre-update mail/Matrix notifications (Issue #24)
├── .sops.yaml # SOPS encryption rules (age key definition)
├── scripts/
│ ├── install-hooks.sh # Installs git hooks for ConfigMap auto-tracking
│ └── hooks/ # Git hooks (pre-commit, post-commit, etc.)
└── docs/
├── README.md # Main deployment guide
├── TASKS.md # Task list & milestones (backlog itself lives in git.lab issues)
├── install.md # Installation instructions
├── ops-configmap-sync.md # ConfigMap syncing with git hooks
└── deployment-guides/ # Detailed guides for specific components (01-07)
```
## Host-Level (non-GitOps) Changes
Almost everything in this repo is reconciled by Flux. `host-config/` is the deliberate
exception: it holds scripts/systemd units meant to run **on the bare Hetzner host itself**
(not as a Kubernetes pod), for things Flux structurally can't reach — e.g. host package
management. There is no SOPS-on-host or Ansible-equivalent mechanism yet; deployment to the
host is manual (`scp`/SSH), and instance-specific values live in a config file on the host
(`/etc/<name>/config`), not hardcoded in the versioned script, so the pattern is reusable
across forks/other communities running this same stack. See
`docs/deployment-guides/07-host-maintenance-notifications.md` for the first (and so far only)
example of this pattern.
## Common Development Commands
### Flux / GitOps Synchronization
```bash
# Force immediate reconciliation (don't wait for 10-min auto-sync)
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization production-apps --with-source
# Check reconciliation status
flux get kustomizations -A
flux get helmreleases -A
# View Flux logs
kubectl logs -n flux-system deployment/source-controller -f
kubectl logs -n flux-system deployment/helm-controller -f
```
### Kubernetes Cluster Status
```bash
# Check pod health in Matrix namespace
kubectl get pods -n matrix
kubectl get pods -n authentik
kubectl get pods -n monitoring
# Detailed pod inspection
kubectl describe pod <pod-name> -n matrix
kubectl logs <pod-name> -n matrix -f
# Check all services and ingresses
kubectl get svc -n matrix
kubectl get ingress -n matrix
```
### Certificate Management (Let's Encrypt / Cert-Manager)
```bash
# View certificate status
kubectl get certificate -n matrix
kubectl get certificaterequest -n matrix
kubectl get challenges -n matrix
# Debug failed certificate issuance
kubectl describe challenge <challenge-name> -n matrix
kubectl logs -n cert-manager deployment/cert-manager -f
# Inspect the issued certificate
kubectl get secret <cert-secret-name> -n matrix -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout
```
### SOPS Secret Editing
SOPS transparently encrypts/decrypts secrets using the `age` key specified in `.sops.yaml`. The environment variable `SOPS_AGE_KEY_FILE` must point to your age private key.
```bash
# Edit an encrypted secret (decrypted for editing, re-encrypted on save)
sops apps/production/custom-configs/mas-secret.yaml
sops apps/authentik/authentik-secret.yaml
# Create a new secret file
sops -i --encrypted-regex '^(data|stringData)$' --input-type yaml --output-type yaml new-secret.yaml
# Decrypt to view
sops -d apps/production/custom-configs/mas-secret.yaml
```
Ensure `~/.age/keys.txt` exists and contains your age private key. See `.devcontainer/devcontainer.json` for setup details.
### Helm Chart Inspection
```bash
# List installed charts
helm list -n matrix
helm list -n authentik
# View rendered chart values
helm get values matrix-stack -n matrix
helm get manifest matrix-stack -n matrix | less
```
### Useful kubectl Shortcuts
```bash
# Port-forward to access services locally
kubectl port-forward -n matrix svc/synapse 8008:8008
# Execute command inside pod (for debugging)
kubectl exec -it <pod-name> -n matrix -- bash
# Stream logs from multiple pods
kubectl logs -n matrix -l app=synapse -f
# Bootstrap a service/bot account via MAS (no registration_shared_secret in this stack)
kubectl exec -it -n matrix deploy/matrix-stack-matrix-authentication-service -- \
mas-cli manage register-user <name> --yes
kubectl exec -it -n matrix deploy/matrix-stack-matrix-authentication-service -- \
mas-cli manage issue-compatibility-token <name>
```
## Architecture & Key Concepts
### FluxCD Reconciliation Flow
1. **Flux watches** `clusters/matrix/` for a FluxRepository resource pointing to this Git repo
2. **Kustomization stages** pull in configurations in order:
- `flux-system` (FluxCD itself)
- `infra-apps` (Namespaces, RBAC, Cert-Manager, HelmRepository sources)
- `production-apps` (Main ESS deployment and related services)
3. **HelmReleases** specify which charts to install and what values to use
4. **ConfigMaps/Secrets** provide values from files in the repo (e.g., custom Synapse config)
5. **Flux auto-reconciles** every 10 minutes, or immediately if Git changes are detected
### Element Server Suite (ESS) Chart Constraints
The ESS Helm chart (v26.4.0) has strict validation and specific quirks:
- **No `config:` blocks for core components** — use ConfigMap overrides instead
- **`serverName` must be at root level**, not nested under `synapse`
- **TLS in Ingress blocks is forbidden** — use `certManager: true` at root to auto-manage certificates
- **`camelCase` for component names**: `elementWeb`, `synapseAdmin`, `matrixAuthenticationService`, etc.
- **OCI HelmRepository only** — the chart is distributed via `oci://ghcr.io/element-hq/ess-helm`, not HTTP
- **Values must pass JSON schema validation** — invalid configs will cause reconciliation failures with cryptic schema errors
### NetworkPolicy Convention
Default-deny-with-explicit-allow across `matrix` and `authentik` namespaces
(`apps/production/networkpolicy.yaml`, `apps/authentik/networkpolicy.yaml`). Every new pod
needs its own explicit ingress-allow rule; NetworkPolicy matches on named **container ports**,
not Service ports — a frequent source of live incidents when a new component is added (wrong
port number/name silently blocks all traffic to it).
### Known Issues & Workarounds
**Issue: Let's Encrypt ACME Race Condition (Error 403 Order's status is processing)**
- Symptom: Certificate provisioning hangs when `elementWeb` and `wellKnownDelegation` are both enabled on the same domain
- Cause: Both request certificates for the same domain simultaneously; Let's Encrypt rejects concurrent requests
- Fix: Set `wellKnownDelegation: enabled: false` and serve `.well-known/matrix/server` via a separate Ingress route or static file
**Issue: HelmChart not ready / stat no such file or directory**
- Cause: Attempting to use a GitRepository source for the ESS chart (it has sub-charts that don't render correctly)
- Fix: Use the OCI HelmRepository source (`oci://ghcr.io/element-hq/ess-helm`) instead
**Issue: Certificate validation failures (No resources found)**
- Cause: Manual Kustomize patches conflict with the Helm chart's built-in certificate management
- Fix: Remove manual patches; rely on `certManager: true` at the root level of HelmRelease values
**Issue: Synapse module can't use asyncio**
- Cause: Synapse runs on Twisted's reactor, not a running asyncio event loop — `asyncio.open_connection`/`asyncio.wait_for` inside a Synapse module (e.g. `clamav_spam_checker.py`) fail immediately with `RuntimeError: no running event loop`, and can silently trigger a fail-open path instead of an obvious crash
- Fix: use `twisted.internet.reactor`/`HostnameEndpoint`/`connectProtocol` + a custom `Protocol` subclass; Twisted `Deferred`s are natively awaitable from `async def` inside Synapse. Standalone processes outside Synapse (e.g. `clamav-http-scanner.py`) don't have this constraint and can use plain sockets/asyncio.
### SOPS Encryption & Key Management
- `.sops.yaml` defines encryption rules (currently using `age` keys)
- Secrets matching the regex in `.sops.yaml` are automatically encrypted when committed
- The age private key (`~/.age/keys.txt`) must be available in your environment for decryption
- In the cluster, Flux decrypts secrets "on the fly" using a secret stored in `flux-system` namespace
To rotate SOPS keys:
```bash
# Regenerate and re-encrypt all secrets
sops updatekeys -y apps/
```
## Development Workflow
### Before Making Changes
1. **Understand dependencies** — check `kustomization.yaml` files to see the order of resource creation
2. **Verify chart schema** — review ESS chart documentation for constraints on the version being used
3. **Test locally if possible** — use `kubectl` port-forwards to verify connectivity before pushing changes
### Making Changes
1. **Edit ConfigMap files directly** — for non-secret customizations (Synapse config, Element Web themes, etc.)
- Changes are auto-tracked by git hooks installed via `./scripts/install-hooks.sh`
2. **Edit secrets with SOPS** — `sops` transparently decrypts/re-encrypts on save
3. **Update HelmRelease values** — modify the `values` section in `element-server-suite.yaml` or reference ConfigMap sources
### After Committing
1. **Flux auto-detects changes** within ~1 minute (or manually trigger with `flux reconcile kustomization production-apps`)
2. **Monitor reconciliation** — watch pod logs and Flux status for errors
3. **Test functionality** — verify services are accessible and functioning as expected
### Git Hooks
After cloning, run:
```bash
./scripts/install-hooks.sh
```
This installs hooks that automatically commit ConfigMap changes to `.gitignore`-like tracking. See `docs/ops-configmap-sync.md` for details.
## Environment Setup
### Local Machine Prerequisites
- `kubectl` — cluster communication
- `flux` — GitOps CLI
- `helm` — chart inspection & debugging
- `sops` & `age` — secret management
- `git` — version control
- age key file at `~/.age/keys.txt` (request from team)
- kubeconfig at `~/.kube/config` (request from team)
### DevContainer (Recommended)
The `.devcontainer/` configuration provides a pre-configured environment:
```bash
# In VS Code: "Reopen in Container"
# Or manually:
docker build -t ess-devcontainer .devcontainer
docker run -it --rm \
-v ~/.kube:/home/vscode/.kube \
-v ~/.age:/home/vscode/.age \
-v ~/.ssh:/home/vscode/.ssh \
-v /var/run/docker.sock:/var/run/docker.sock \
ess-devcontainer
```
DevContainer includes:
- All required CLI tools (kubectl, flux, helm, sops, age, git, docker)
- VS Code extensions for YAML, Kubernetes, Helm
- Proper environment variables (`KUBECONFIG`, `SOPS_AGE_KEY_FILE`)
- Git hooks pre-installed
## Troubleshooting Checklist
- **Pod not starting?** → `kubectl describe pod <name> -n matrix` (check events)
- **Image pull failures?** → Check HelmRelease status: `kubectl get helmrelease -n matrix`
- **Secret not found?** → Verify SOPS decryption: `sops -d <secret.sops.yaml>` (must output valid YAML)
- **Certificate stuck?** → `kubectl describe certificate <name> -n matrix` (check for ACME errors)
- **Config validation error?** → Inspect HelmRelease status: `kubectl describe helmrelease <name> -n matrix` (JSON schema error message)
- **Cluster unreachable?** → Verify kubeconfig: `kubectl get nodes` (must connect to K3s)
- **NetworkPolicy blocking a new pod?** → Check it matches on container port name, not Service port
## Resources & References
- **README.md** — High-level overview and architecture
- **docs/TASKS.md** — Task backlog, milestones, and priority list (open backlog lives in [git.lab issues](https://git.lab/axion1337.chat/axion1337.chat-gitops/-/issues); the closed Gitea issues only point there)
- **docs/deployment-guides/** — Detailed setup guides for specific components (01-07)
- **docs/ops-configmap-sync.md** — Git hook configuration and auto-sync behavior
- **ESS Chart Docs** — `https://github.com/element-hq/ess-helm` (official Helm chart repository)
- **FluxCD Docs** — `https://fluxcd.io/docs/` (GitOps reconciliation & Kustomization)
- **Matrix Spec** — `https://spec.matrix.org/` (Matrix protocol specification)
---
name: karpathy-guidelines
description: Behavioral guidelines to reduce common LLM coding mistakes. Use when writing, reviewing, or refactoring code to avoid overcomplication, make surgical changes, surface assumptions, and define verifiable success criteria.
license: MIT
---
# Karpathy Guidelines
Behavioral guidelines to reduce common LLM coding mistakes, derived from [Andrej Karpathy's observations](https://x.com/karpathy/status/2015883857489522876) on LLM coding pitfalls.
**Tradeoff:** These guidelines bias toward caution over speed. For trivial tasks, use judgment.
## 1. Think Before Coding
**Don't assume. Don't hide confusion. Surface tradeoffs.**
Before implementing:
- State your assumptions explicitly. If uncertain, ask.
- If multiple interpretations exist, present them - don't pick silently.
- If a simpler approach exists, say so. Push back when warranted.
- If something is unclear, stop. Name what's confusing. Ask.
## 2. Simplicity First
**Minimum code that solves the problem. Nothing speculative.**
- No features beyond what was asked.
- No abstractions for single-use code.
- No "flexibility" or "configurability" that wasn't requested.
- No error handling for impossible scenarios.
- If you write 200 lines and it could be 50, rewrite it.
Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify.
## 3. Surgical Changes
**Touch only what you must. Clean up only your own mess.**
When editing existing code:
- Don't "improve" adjacent code, comments, or formatting.
- Don't refactor things that aren't broken.
- Match existing style, even if you'd do it differently.
- If you notice unrelated dead code, mention it - don't delete it.
When your changes create orphans:
- Remove imports/variables/functions that YOUR changes made unused.
- Don't remove pre-existing dead code unless asked.
The test: Every changed line should trace directly to the user's request.
## 4. Goal-Driven Execution
**Define success criteria. Loop until verified.**
Transform tasks into verifiable goals:
- "Add validation" → "Write tests for invalid inputs, then make them pass"
- "Fix the bug" → "Write a test that reproduces it, then make it pass"
- "Refactor X" → "Ensure tests pass before and after"
For multi-step tasks, state a brief plan:
```
1. [Step] → verify: [check]
2. [Step] → verify: [check]
3. [Step] → verify: [check]
```
Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification.
Read AGENTS.md — the canonical instruction file for this repository. All rules live there.