axion1337.chat-gitops/AGENTS.md

# AGENTS.md — axion1337.chat-gitops

> **Die Gruppenregeln sind kanonisch im `management`-Repo:**
> [`AGENTS.md`](https://git.lab/axion1337.chat/management/-/blob/main/AGENTS.md)
> — von außerhalb des Labs über den Gitea-Mirror lesbar:
> `https://rohana.axion1337.de/sorb/management`. Dort stehen Repo-Topologie und
> Mirror-Regeln, das Kanban-Framework (Status-Labels, WIP-Limit 2, ADR-Pflicht —
> auch für dauerhafte Regel-Ausnahmen), Deploy-Übergabe und AAR-Verfahren,
> Secrets-Handhabung und die Karpathy-Leitlinien. Sie gelten für **jede** Session
> in diesem Repo. Hier steht nur, was für dieses Repository zusätzlich gilt.

## Overview

This is a **GitOps-based Kubernetes deployment** of **Element Server Suite (ESS Community v26.4.0)**, a complete Matrix homeserver stack. The repository contains Infrastructure-as-Code using **FluxCD** for GitOps synchronization, with encryption (SOPS/age), service mesh (Traefik), certificate management (Cert-Manager), and auxiliary services like Authentik, TURN/coturn, Draupnir (moderation), ClamAV (content scanning), and Grafana monitoring.

**Key Stack Components:**
- **K3s**: Lightweight Kubernetes distribution running on Hetzner Cloud
- **FluxCD**: GitOps controller that watches this repository and auto-syncs changes
- **ESS (Matrix Umbrella Chart v26.4.0)**: Synapse, MAS (Matrix Authentication Service), ElementWeb, MatrixRTC
- **Authentik**: OIDC-based identity provider for centralized authentication, deployed both via HelmRelease and declarative Blueprints (`apps/authentik/authentik-blueprints.yaml`) for flows/OIDC-provider config that would otherwise only exist as manual admin-UI clicks
- **Traefik**: Ingress controller (built into K3s) for routing HTTP/HTTPS traffic
- **Cert-Manager**: Automatic TLS certificate provisioning from Let's Encrypt
- **SOPS + age**: Transparent encryption/decryption of secrets in Git
- **Monitoring**: Grafana Alloy (agent), Prometheus (metrics), Loki (logs)
- **coturn**: TURN/STUN server for WebRTC audio/video calls, with monthly automated shared-secret rotation via CronJob + PR workflow
- **Draupnir**: Matrix moderation bot (community successor to Mjolnir), ban lists/policy rooms
- **ClamAV**: Content scanning — a Synapse module for unencrypted-room uploads, plus a standalone `clamav-http-scanner` service that a patched Element Web client (ThreadNet-Web) calls both on send and on receive, extending coverage to encrypted rooms/DMs
- **NetworkPolicies**: default-deny-with-explicit-allow across `matrix` and `authentik` namespaces
- **`host-config/`**: the one part of this repo that is deliberately **not** managed by Flux/GitOps — see "Host-Level (non-GitOps) Changes" below

## Repo Topology (since 2026-07-31)

Canonical repo is **`git.lab/axion1337.chat/axion1337.chat-gitops`** (homelab GitLab,
resolvable only inside the lab) — all pushes go there; a push-mirror updates the Gitea
copy on `rohana.axion1337.de`, which remains the **Flux source** (the cluster pulls from
Gitea; the mirror delivers). **Never push directly to Gitea** for this repo — the mirror
force-overwrites divergent state. The same rule applies to ThreadNet-Web, threadnet-call,
thread-net-git, threadnet-operating and (since 2026-08-01) `management` (the former
`Backlogs` repo, renamed in the PM-framework restructuring — ADRs/vision/roadmap live
there). Since 2026-08-02 the wiki lives on git.lab too, so **no repo is authored on
Gitea any more**; the one process that still *writes* there is the TURN rotation
(see below).

**Why Gitea is the Flux source, and why that is not a leftover.** git.lab holds the
blueprints; Gitea holds a copy the cluster can reach without the lab being up. That
separation is deliberate: the Hetzner cluster must be buildable and re-deployable when
the homelab is offline, on holiday, or mid-rebuild — it therefore must not depend on a
host that only answers inside the lab. **Do not "fix" the Flux source to point at
git.lab**: it would look tidier and would couple production availability to the lab,
which is exactly what this split avoids.

**Issues live on git.lab** (migrated 2026-08-01, gitops#48): the old Gitea issues are
closed with a pointer to their GitLab counterpart. ⚠️ gitops issue numbers **shifted**
in the migration (Gitea had gaps from PRs; e.g. Gitea#48 → GitLab#46) — old references
"gitops#N" in commits/docs mean the **Gitea** number; the authoritative mapping is the
migration footer in each GitLab issue. Releases stay on Gitea (public download path).
**No exceptions left**: deploy-handover issues used to stay on the Gitea
`sorb/management` tracker because CFGMON had no route to git.lab. The site-to-site VPN
(ADR-0004) removed that reason, and on 2026-08-02 they moved (LABNET-03) — the Gitea
tracker is empty and the form now lives as a GitLab issue template in the management
repo. Issues live on git.lab, without exception.

**Wiki** (migrated 2026-08-02): the 15 pages now live in this project's GitLab wiki
(`…-gitops.wiki.git`, *Wiki* tab). ⚠️ The **`wiki` branch in this repo is NOT the wiki** —
it is a stale May snapshot of `docs/`; don't edit or trust it.

⚠️ **`wiki.lab` is gone** (2026-08-15). The Docusaurus site that aggregated the doc
sources was replaced by **Wiki.js inside this stack** (ADR-0014, `wiki.axion1337.chat`),
which holds the operations and user handbooks and is edited in its own UI rather than
aggregated read-only from source repos. `homelab/docs` is deliberately out of that
scope — sorb's homelab documentation is not part of the platform.

**The one write that still lands on Gitea**: the monthly TURN-rotation CronJob runs in
the cluster (no route to git.lab) and pushes its rotation branch to Gitea. Never merge
that PR on Gitea — the rotation has to travel back through git.lab, otherwise the next
mirror push overwrites it and Flux silently re-applies the *old* shared secret.

**This is automated — do not do it by hand.** The scheduled job `canonize_rotation` in
`.gitlab-ci.yml` runs daily on git.lab, picks up any `turn-secret-rotation-*` branch
from Gitea that is not yet in `main`, merges it, and pushes to git.lab; the mirror
carries it back and Gitea auto-closes the PR. Once merged, the branch exists only on
Gitea, so the next mirror run deletes it — no cleanup needed.

If the job fails (merge conflict, or the SOPS check finds an unencrypted secret file),
it fails **loudly and changes nothing** — the pipeline stays red until someone looks.
That red pipeline is the alarm; there is no separate reminder. Manual fallback, should
it ever be needed: fetch the branch, merge into main locally, push to git.lab (worked
example: 2026-08-01, commit `717638d`). If the mirror lags, force it via the GitLab API
`POST /projects/<id>/remote_mirrors/<mirror_id>/sync`.

## Repository Structure

```
gitops/
├── .gitlab-ci.yml            # Lab-CI verify job (YAML validation + SOPS-encryption check)
├── clusters/matrix/          # Flux GitRepository definition; entry point for reconciliation
├── apps/
│   ├── base/
│   │   ├── infra/            # Core infrastructure (Cert-Manager, Namespaces, etc.)
│   │   └── matrix/           # HelmRepository definition for ESS OCI chart
│   ├── production/           # Main ESS deployment
│   │   ├── element-server-suite.yaml     # HelmRelease (ESS chart v26.4.0)
│   │   ├── custom-configs/               # Overrides & custom configurations
│   │   │   ├── synapse-values.yaml       # Synapse customizations (ConfigMap)
│   │   │   ├── element-values.yaml       # ElementWeb customizations (ConfigMap)
│   │   │   └── mas-secret.yaml           # MAS secrets (encrypted with SOPS)
│   │   ├── cert-issuer.yaml              # Let's Encrypt ClusterIssuer
│   │   ├── apex-ingress.yaml              # Apex-domain IngressRoutes (Element Web, /_scan, etc.)
│   │   ├── matrix-postgres-auth.yaml     # PostgreSQL credentials
│   │   ├── coturn.yaml / coturn-secret.yaml / synapse-turn-secret.yaml
│   │   ├── turn-secret-rotation.yaml     # Monthly CronJob, rotates coturn shared secret via PR
│   │   ├── draupnir.yaml / draupnir-pvc.yaml / draupnir-secret.yaml
│   │   ├── clamav.yaml / clamav-pvc.yaml / clamav_spam_checker.py   # Synapse-side scan module
│   │   ├── clamav-http-scanner.py / -Dockerfile / .yaml             # Client-side scan service
│   │   ├── synapse-backup.yaml / synapse-backup-secret.yaml
│   │   └── networkpolicy.yaml            # Default-deny + explicit allow rules
│   ├── authentik/            # Identity Provider (separate namespace)
│   │   ├── authentik.yaml                # HelmRelease
│   │   ├── authentik-blueprints.yaml     # Flows/OIDC-provider as declarative code
│   │   ├── helm-repo.yaml                # HelmRepository source
│   │   ├── ingress.yaml                  # Ingress route
│   │   ├── networkpolicy.yaml
│   │   └── authentik-secret.yaml         # Secrets (admin password, OIDC client secret, etc.)
│   └── monitoring/           # Observability (Alloy, kube-state-metrics, node-exporter)
│       ├── alloy-config.yaml             # Grafana Alloy configuration
│       └── kube-state-metrics.yaml       # K8s metrics exporter
├── host-config/               # Host-level (non-GitOps) config, see below
│   └── maintenance-notify/    # systemd timer: pre-update mail/Matrix notifications (Issue #24)
├── .sops.yaml                # SOPS encryption rules (age key definition)
├── scripts/
│   ├── install-hooks.sh      # Installs git hooks for ConfigMap auto-tracking
│   └── hooks/                # Git hooks (pre-commit, post-commit, etc.)
└── docs/
    ├── README.md             # Main deployment guide
    ├── TASKS.md               # Task list & milestones (backlog itself lives in git.lab issues)
    ├── install.md             # Installation instructions
    ├── ops-configmap-sync.md # ConfigMap syncing with git hooks
    └── deployment-guides/    # Detailed guides for specific components (01-07)
```

## Host-Level (non-GitOps) Changes

Almost everything in this repo is reconciled by Flux. `host-config/` is the deliberate
exception: it holds scripts/systemd units meant to run **on the bare Hetzner host itself**
(not as a Kubernetes pod), for things Flux structurally can't reach — e.g. host package
management. There is no SOPS-on-host or Ansible-equivalent mechanism yet; deployment to the
host is manual (`scp`/SSH), and instance-specific values live in a config file on the host
(`/etc/<name>/config`), not hardcoded in the versioned script, so the pattern is reusable
across forks/other communities running this same stack. See
`docs/deployment-guides/07-host-maintenance-notifications.md` for the first (and so far only)
example of this pattern.

## Common Development Commands

### Flux / GitOps Synchronization

```bash
# Force immediate reconciliation (don't wait for 10-min auto-sync)
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization production-apps --with-source

# Check reconciliation status
flux get kustomizations -A
flux get helmreleases -A

# View Flux logs
kubectl logs -n flux-system deployment/source-controller -f
kubectl logs -n flux-system deployment/helm-controller -f
```

### Kubernetes Cluster Status

```bash
# Check pod health in Matrix namespace
kubectl get pods -n matrix
kubectl get pods -n authentik
kubectl get pods -n monitoring

# Detailed pod inspection
kubectl describe pod <pod-name> -n matrix
kubectl logs <pod-name> -n matrix -f

# Check all services and ingresses
kubectl get svc -n matrix
kubectl get ingress -n matrix
```

### Certificate Management (Let's Encrypt / Cert-Manager)

```bash
# View certificate status
kubectl get certificate -n matrix
kubectl get certificaterequest -n matrix
kubectl get challenges -n matrix

# Debug failed certificate issuance
kubectl describe challenge <challenge-name> -n matrix
kubectl logs -n cert-manager deployment/cert-manager -f

# Inspect the issued certificate
kubectl get secret <cert-secret-name> -n matrix -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout
```

### SOPS Secret Editing

SOPS transparently encrypts/decrypts secrets using the `age` key specified in `.sops.yaml`. The environment variable `SOPS_AGE_KEY_FILE` must point to your age private key.

```bash
# Edit an encrypted secret (decrypted for editing, re-encrypted on save)
sops apps/production/custom-configs/mas-secret.yaml
sops apps/authentik/authentik-secret.yaml

# Create a new secret file
sops -i --encrypted-regex '^(data|stringData)$' --input-type yaml --output-type yaml new-secret.yaml

# Decrypt to view
sops -d apps/production/custom-configs/mas-secret.yaml
```

Ensure `~/.age/keys.txt` exists and contains your age private key. See `.devcontainer/devcontainer.json` for setup details.

### Helm Chart Inspection

```bash
# List installed charts
helm list -n matrix
helm list -n authentik

# View rendered chart values
helm get values matrix-stack -n matrix
helm get manifest matrix-stack -n matrix | less
```

### Useful kubectl Shortcuts

```bash
# Port-forward to access services locally
kubectl port-forward -n matrix svc/synapse 8008:8008

# Execute command inside pod (for debugging)
kubectl exec -it <pod-name> -n matrix -- bash

# Stream logs from multiple pods
kubectl logs -n matrix -l app=synapse -f

# Bootstrap a service/bot account via MAS (no registration_shared_secret in this stack)
kubectl exec -it -n matrix deploy/matrix-stack-matrix-authentication-service -- \
  mas-cli manage register-user <name> --yes
kubectl exec -it -n matrix deploy/matrix-stack-matrix-authentication-service -- \
  mas-cli manage issue-compatibility-token <name>
```

## Architecture & Key Concepts

### FluxCD Reconciliation Flow

1. **Flux watches** `clusters/matrix/` for a FluxRepository resource pointing to this Git repo
2. **Kustomization stages** pull in configurations in order:
   - `flux-system` (FluxCD itself)
   - `infra-apps` (Namespaces, RBAC, Cert-Manager, HelmRepository sources)
   - `production-apps` (Main ESS deployment and related services)
3. **HelmReleases** specify which charts to install and what values to use
4. **ConfigMaps/Secrets** provide values from files in the repo (e.g., custom Synapse config)
5. **Flux auto-reconciles** every 10 minutes, or immediately if Git changes are detected

### Element Server Suite (ESS) Chart Constraints

The ESS Helm chart (v26.4.0) has strict validation and specific quirks:

- **No `config:` blocks for core components** — use ConfigMap overrides instead
- **`serverName` must be at root level**, not nested under `synapse`
- **TLS in Ingress blocks is forbidden** — use `certManager: true` at root to auto-manage certificates
- **`camelCase` for component names**: `elementWeb`, `synapseAdmin`, `matrixAuthenticationService`, etc.
- **OCI HelmRepository only** — the chart is distributed via `oci://ghcr.io/element-hq/ess-helm`, not HTTP
- **Values must pass JSON schema validation** — invalid configs will cause reconciliation failures with cryptic schema errors

### NetworkPolicy Convention

Default-deny-with-explicit-allow across `matrix` and `authentik` namespaces
(`apps/production/networkpolicy.yaml`, `apps/authentik/networkpolicy.yaml`). Every new pod
needs its own explicit ingress-allow rule; NetworkPolicy matches on named **container ports**,
not Service ports — a frequent source of live incidents when a new component is added (wrong
port number/name silently blocks all traffic to it).

### Known Issues & Workarounds

**Issue: Let's Encrypt ACME Race Condition (Error 403 Order's status is processing)**
- Symptom: Certificate provisioning hangs when `elementWeb` and `wellKnownDelegation` are both enabled on the same domain
- Cause: Both request certificates for the same domain simultaneously; Let's Encrypt rejects concurrent requests
- Fix: Set `wellKnownDelegation: enabled: false` and serve `.well-known/matrix/server` via a separate Ingress route or static file

**Issue: HelmChart not ready / stat no such file or directory**
- Cause: Attempting to use a GitRepository source for the ESS chart (it has sub-charts that don't render correctly)
- Fix: Use the OCI HelmRepository source (`oci://ghcr.io/element-hq/ess-helm`) instead

**Issue: Certificate validation failures (No resources found)**
- Cause: Manual Kustomize patches conflict with the Helm chart's built-in certificate management
- Fix: Remove manual patches; rely on `certManager: true` at the root level of HelmRelease values

**Issue: Synapse module can't use asyncio**
- Cause: Synapse runs on Twisted's reactor, not a running asyncio event loop — `asyncio.open_connection`/`asyncio.wait_for` inside a Synapse module (e.g. `clamav_spam_checker.py`) fail immediately with `RuntimeError: no running event loop`, and can silently trigger a fail-open path instead of an obvious crash
- Fix: use `twisted.internet.reactor`/`HostnameEndpoint`/`connectProtocol` + a custom `Protocol` subclass; Twisted `Deferred`s are natively awaitable from `async def` inside Synapse. Standalone processes outside Synapse (e.g. `clamav-http-scanner.py`) don't have this constraint and can use plain sockets/asyncio.

### SOPS Encryption & Key Management

- `.sops.yaml` defines encryption rules (currently using `age` keys)
- Secrets matching the regex in `.sops.yaml` are automatically encrypted when committed
- The age private key (`~/.age/keys.txt`) must be available in your environment for decryption
- In the cluster, Flux decrypts secrets "on the fly" using a secret stored in `flux-system` namespace

To rotate SOPS keys:
```bash
# Regenerate and re-encrypt all secrets
sops updatekeys -y apps/
```

## Development Workflow

### Before Making Changes

1. **Understand dependencies** — check `kustomization.yaml` files to see the order of resource creation
2. **Verify chart schema** — review ESS chart documentation for constraints on the version being used
3. **Test locally if possible** — use `kubectl` port-forwards to verify connectivity before pushing changes

### Making Changes

1. **Edit ConfigMap files directly** — for non-secret customizations (Synapse config, Element Web themes, etc.)
   - Changes are auto-tracked by git hooks installed via `./scripts/install-hooks.sh`
2. **Edit secrets with SOPS** — `sops` transparently decrypts/re-encrypts on save
3. **Update HelmRelease values** — modify the `values` section in `element-server-suite.yaml` or reference ConfigMap sources

### After Committing

1. **Flux auto-detects changes** within ~1 minute (or manually trigger with `flux reconcile kustomization production-apps`)
2. **Monitor reconciliation** — watch pod logs and Flux status for errors
3. **Test functionality** — verify services are accessible and functioning as expected

### Git Hooks

After cloning, run:
```bash
./scripts/install-hooks.sh
```

This installs hooks that automatically commit ConfigMap changes to `.gitignore`-like tracking. See `docs/ops-configmap-sync.md` for details.

## Environment Setup

### Local Machine Prerequisites

- `kubectl` — cluster communication
- `flux` — GitOps CLI
- `helm` — chart inspection & debugging
- `sops` & `age` — secret management
- `git` — version control
- age key file at `~/.age/keys.txt` (request from team)
- kubeconfig at `~/.kube/config` (request from team)

### DevContainer (Recommended)

The `.devcontainer/` configuration provides a pre-configured environment:

```bash
# In VS Code: "Reopen in Container"
# Or manually:
docker build -t ess-devcontainer .devcontainer
docker run -it --rm \
  -v ~/.kube:/home/vscode/.kube \
  -v ~/.age:/home/vscode/.age \
  -v ~/.ssh:/home/vscode/.ssh \
  -v /var/run/docker.sock:/var/run/docker.sock \
  ess-devcontainer
```

DevContainer includes:
- All required CLI tools (kubectl, flux, helm, sops, age, git, docker)
- VS Code extensions for YAML, Kubernetes, Helm
- Proper environment variables (`KUBECONFIG`, `SOPS_AGE_KEY_FILE`)
- Git hooks pre-installed

## Troubleshooting Checklist

- **Pod not starting?** → `kubectl describe pod <name> -n matrix` (check events)
- **Image pull failures?** → Check HelmRelease status: `kubectl get helmrelease -n matrix`
- **Secret not found?** → Verify SOPS decryption: `sops -d <secret.sops.yaml>` (must output valid YAML)
- **Certificate stuck?** → `kubectl describe certificate <name> -n matrix` (check for ACME errors)
- **Config validation error?** → Inspect HelmRelease status: `kubectl describe helmrelease <name> -n matrix` (JSON schema error message)
- **Cluster unreachable?** → Verify kubeconfig: `kubectl get nodes` (must connect to K3s)
- **NetworkPolicy blocking a new pod?** → Check it matches on container port name, not Service port

## Resources & References

- **README.md** — High-level overview and architecture
- **docs/TASKS.md** — Task backlog, milestones, and priority list (open backlog lives in [git.lab issues](https://git.lab/axion1337.chat/axion1337.chat-gitops/-/issues); the closed Gitea issues only point there)
- **docs/deployment-guides/** — Detailed setup guides for specific components (01-07)
- **docs/ops-configmap-sync.md** — Git hook configuration and auto-sync behavior
- **ESS Chart Docs** — `https://github.com/element-hq/ess-helm` (official Helm chart repository)
- **FluxCD Docs** — `https://fluxcd.io/docs/` (GitOps reconciliation & Kustomization)
- **Matrix Spec** — `https://spec.matrix.org/` (Matrix protocol specification)

---
name: karpathy-guidelines
description: Behavioral guidelines to reduce common LLM coding mistakes. Use when writing, reviewing, or refactoring code to avoid overcomplication, make surgical changes, surface assumptions, and define verifiable success criteria.
license: MIT
---

# Karpathy Guidelines

Behavioral guidelines to reduce common LLM coding mistakes, derived from [Andrej Karpathy's observations](https://x.com/karpathy/status/2015883857489522876) on LLM coding pitfalls.

**Tradeoff:** These guidelines bias toward caution over speed. For trivial tasks, use judgment.

## 1. Think Before Coding

**Don't assume. Don't hide confusion. Surface tradeoffs.**

Before implementing:
- State your assumptions explicitly. If uncertain, ask.
- If multiple interpretations exist, present them - don't pick silently.
- If a simpler approach exists, say so. Push back when warranted.
- If something is unclear, stop. Name what's confusing. Ask.

## 2. Simplicity First

**Minimum code that solves the problem. Nothing speculative.**

- No features beyond what was asked.
- No abstractions for single-use code.
- No "flexibility" or "configurability" that wasn't requested.
- No error handling for impossible scenarios.
- If you write 200 lines and it could be 50, rewrite it.

Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify.

## 3. Surgical Changes

**Touch only what you must. Clean up only your own mess.**

When editing existing code:
- Don't "improve" adjacent code, comments, or formatting.
- Don't refactor things that aren't broken.
- Match existing style, even if you'd do it differently.
- If you notice unrelated dead code, mention it - don't delete it.

When your changes create orphans:
- Remove imports/variables/functions that YOUR changes made unused.
- Don't remove pre-existing dead code unless asked.

The test: Every changed line should trace directly to the user's request.

## 4. Goal-Driven Execution

**Define success criteria. Loop until verified.**

Transform tasks into verifiable goals:
- "Add validation" → "Write tests for invalid inputs, then make them pass"
- "Fix the bug" → "Write a test that reproduces it, then make it pass"
- "Refactor X" → "Ensure tests pass before and after"

For multi-step tasks, state a brief plan:
```
1. [Step] → verify: [check]
2. [Step] → verify: [check]
3. [Step] → verify: [check]
```

Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification.