axion1337.chat-gitops/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Cross-project working conventions are canonical in the management repo's CLAUDE.md (readable from anywhere via the Gitea mirror https://rohana.axion1337.de/sorb/management): repo topology & mirror rules, the Kanban framework (status labels, WIP limit 2, ADR duty — including for permanent rule exceptions), deploy-handover/AAR procedures, secrets handling, and the Karpathy guidelines. Every agent session must follow it; this file covers only what is specific to this repository.

Overview

This is a GitOps-based Kubernetes deployment of Element Server Suite (ESS Community v26.4.0), a complete Matrix homeserver stack. The repository contains Infrastructure-as-Code using FluxCD for GitOps synchronization, with encryption (SOPS/age), service mesh (Traefik), certificate management (Cert-Manager), and auxiliary services like Authentik, TURN/coturn, Draupnir (moderation), ClamAV (content scanning), and Grafana monitoring.

Key Stack Components:

• K3s: Lightweight Kubernetes distribution running on Hetzner Cloud
• FluxCD: GitOps controller that watches this repository and auto-syncs changes
• ESS (Matrix Umbrella Chart v26.4.0): Synapse, MAS (Matrix Authentication Service), ElementWeb, MatrixRTC
• Authentik: OIDC-based identity provider for centralized authentication, deployed both via HelmRelease and declarative Blueprints (apps/authentik/authentik-blueprints.yaml) for flows/OIDC-provider config that would otherwise only exist as manual admin-UI clicks
• Traefik: Ingress controller (built into K3s) for routing HTTP/HTTPS traffic
• Cert-Manager: Automatic TLS certificate provisioning from Let's Encrypt
• SOPS + age: Transparent encryption/decryption of secrets in Git
• Monitoring: Grafana Alloy (agent), Prometheus (metrics), Loki (logs)
• coturn: TURN/STUN server for WebRTC audio/video calls, with monthly automated shared-secret rotation via CronJob + PR workflow
• Draupnir: Matrix moderation bot (community successor to Mjolnir), ban lists/policy rooms
• ClamAV: Content scanning — a Synapse module for unencrypted-room uploads, plus a standalone clamav-http-scanner service that a patched Element Web client (ThreadNet-Web) calls both on send and on receive, extending coverage to encrypted rooms/DMs
• NetworkPolicies: default-deny-with-explicit-allow across matrix and authentik namespaces
• host-config/: the one part of this repo that is deliberately not managed by Flux/GitOps — see "Host-Level (non-GitOps) Changes" below

Repo Topology (since 2026-07-31)

Canonical repo is git.lab/axion1337.chat/axion1337.chat-gitops (homelab GitLab, resolvable only inside the lab) — all pushes go there; a push-mirror updates the Gitea copy on rohana.axion1337.de, which remains the Flux source (the cluster pulls from Gitea; the mirror delivers). Never push directly to Gitea for this repo — the mirror force-overwrites divergent state. The same rule applies to ThreadNet-Web, threadnet-call, thread-net-git, threadnet-operating and (since 2026-08-01) management (the former Backlogs repo, renamed in the PM-framework restructuring — ADRs/vision/roadmap live there). Since 2026-08-02 the wiki lives on git.lab too, so no repo is authored on Gitea any more; the one process that still writes there is the TURN rotation (see below).

Why Gitea is the Flux source, and why that is not a leftover. git.lab holds the blueprints; Gitea holds a copy the cluster can reach without the lab being up. That separation is deliberate: the Hetzner cluster must be buildable and re-deployable when the homelab is offline, on holiday, or mid-rebuild — it therefore must not depend on a host that only answers inside the lab. Do not "fix" the Flux source to point at git.lab: it would look tidier and would couple production availability to the lab, which is exactly what this split avoids.

Issues live on git.lab (migrated 2026-08-01, gitops#48): the old Gitea issues are closed with a pointer to their GitLab counterpart. ⚠️ gitops issue numbers shifted in the migration (Gitea had gaps from PRs; e.g. Gitea#48 → GitLab#46) — old references "gitops#N" in commits/docs mean the Gitea number; the authoritative mapping is the migration footer in each GitLab issue. Releases stay on Gitea (public download path). No exceptions left: deploy-handover issues used to stay on the Gitea sorb/management tracker because CFGMON had no route to git.lab. The site-to-site VPN (ADR-0004) removed that reason, and on 2026-08-02 they moved (LABNET-03) — the Gitea tracker is empty and the form now lives as a GitLab issue template in the management repo. Issues live on git.lab, without exception.

Wiki (migrated 2026-08-02): the 15 pages now live in this project's GitLab wiki (…-gitops.wiki.git, Wiki tab). ⚠️ The wiki branch in this repo is NOT the wiki — it is a stale May snapshot of docs/; don't edit or trust it. All doc sources (platform wiki, homelab/docs, management) are also served side by side as one Docusaurus site at wiki.lab, configured in git.lab/homelab/wiki — content is pulled at build time, so edits always belong in the source repo.

The one write that still lands on Gitea: the monthly TURN-rotation CronJob runs in the cluster (no route to git.lab) and pushes its rotation branch to Gitea. Never merge that PR on Gitea — the rotation has to travel back through git.lab, otherwise the next mirror push overwrites it and Flux silently re-applies the old shared secret.

This is automated — do not do it by hand. The scheduled job canonize_rotation in .gitlab-ci.yml runs daily on git.lab, picks up any turn-secret-rotation-* branch from Gitea that is not yet in main, merges it, and pushes to git.lab; the mirror carries it back and Gitea auto-closes the PR. Once merged, the branch exists only on Gitea, so the next mirror run deletes it — no cleanup needed.

If the job fails (merge conflict, or the SOPS check finds an unencrypted secret file), it fails loudly and changes nothing — the pipeline stays red until someone looks. That red pipeline is the alarm; there is no separate reminder. Manual fallback, should it ever be needed: fetch the branch, merge into main locally, push to git.lab (worked example: 2026-08-01, commit 717638d). If the mirror lags, force it via the GitLab API POST /projects/<id>/remote_mirrors/<mirror_id>/sync.

Repository Structure

gitops/
├── .gitlab-ci.yml            # Lab-CI verify job (YAML validation + SOPS-encryption check)
├── clusters/matrix/          # Flux GitRepository definition; entry point for reconciliation
├── apps/
│   ├── base/
│   │   ├── infra/            # Core infrastructure (Cert-Manager, Namespaces, etc.)
│   │   └── matrix/           # HelmRepository definition for ESS OCI chart
│   ├── production/           # Main ESS deployment
│   │   ├── element-server-suite.yaml     # HelmRelease (ESS chart v26.4.0)
│   │   ├── custom-configs/               # Overrides & custom configurations
│   │   │   ├── synapse-values.yaml       # Synapse customizations (ConfigMap)
│   │   │   ├── element-values.yaml       # ElementWeb customizations (ConfigMap)
│   │   │   └── mas-secret.yaml           # MAS secrets (encrypted with SOPS)
│   │   ├── cert-issuer.yaml              # Let's Encrypt ClusterIssuer
│   │   ├── apex-ingress.yaml              # Apex-domain IngressRoutes (Element Web, /_scan, etc.)
│   │   ├── matrix-postgres-auth.yaml     # PostgreSQL credentials
│   │   ├── coturn.yaml / coturn-secret.yaml / synapse-turn-secret.yaml
│   │   ├── turn-secret-rotation.yaml     # Monthly CronJob, rotates coturn shared secret via PR
│   │   ├── draupnir.yaml / draupnir-pvc.yaml / draupnir-secret.yaml
│   │   ├── clamav.yaml / clamav-pvc.yaml / clamav_spam_checker.py   # Synapse-side scan module
│   │   ├── clamav-http-scanner.py / -Dockerfile / .yaml             # Client-side scan service
│   │   ├── synapse-backup.yaml / synapse-backup-secret.yaml
│   │   └── networkpolicy.yaml            # Default-deny + explicit allow rules
│   ├── authentik/            # Identity Provider (separate namespace)
│   │   ├── authentik.yaml                # HelmRelease
│   │   ├── authentik-blueprints.yaml     # Flows/OIDC-provider as declarative code
│   │   ├── helm-repo.yaml                # HelmRepository source
│   │   ├── ingress.yaml                  # Ingress route
│   │   ├── networkpolicy.yaml
│   │   └── authentik-secret.yaml         # Secrets (admin password, OIDC client secret, etc.)
│   └── monitoring/           # Observability (Alloy, kube-state-metrics, node-exporter)
│       ├── alloy-config.yaml             # Grafana Alloy configuration
│       └── kube-state-metrics.yaml       # K8s metrics exporter
├── host-config/               # Host-level (non-GitOps) config, see below
│   └── maintenance-notify/    # systemd timer: pre-update mail/Matrix notifications (Issue #24)
├── .sops.yaml                # SOPS encryption rules (age key definition)
├── scripts/
│   ├── install-hooks.sh      # Installs git hooks for ConfigMap auto-tracking
│   └── hooks/                # Git hooks (pre-commit, post-commit, etc.)
└── docs/
    ├── README.md             # Main deployment guide
    ├── TASKS.md               # Task list & milestones (backlog itself lives in git.lab issues)
    ├── install.md             # Installation instructions
    ├── ops-configmap-sync.md # ConfigMap syncing with git hooks
    └── deployment-guides/    # Detailed guides for specific components (01-07)

Host-Level (non-GitOps) Changes

Almost everything in this repo is reconciled by Flux. host-config/ is the deliberate exception: it holds scripts/systemd units meant to run on the bare Hetzner host itself (not as a Kubernetes pod), for things Flux structurally can't reach — e.g. host package management. There is no SOPS-on-host or Ansible-equivalent mechanism yet; deployment to the host is manual (scp/SSH), and instance-specific values live in a config file on the host (/etc/<name>/config), not hardcoded in the versioned script, so the pattern is reusable across forks/other communities running this same stack. See docs/deployment-guides/07-host-maintenance-notifications.md for the first (and so far only) example of this pattern.

Common Development Commands

Flux / GitOps Synchronization

# Force immediate reconciliation (don't wait for 10-min auto-sync)
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization production-apps --with-source

# Check reconciliation status
flux get kustomizations -A
flux get helmreleases -A

# View Flux logs
kubectl logs -n flux-system deployment/source-controller -f
kubectl logs -n flux-system deployment/helm-controller -f

Kubernetes Cluster Status

# Check pod health in Matrix namespace
kubectl get pods -n matrix
kubectl get pods -n authentik
kubectl get pods -n monitoring

# Detailed pod inspection
kubectl describe pod <pod-name> -n matrix
kubectl logs <pod-name> -n matrix -f

# Check all services and ingresses
kubectl get svc -n matrix
kubectl get ingress -n matrix

Certificate Management (Let's Encrypt / Cert-Manager)

# View certificate status
kubectl get certificate -n matrix
kubectl get certificaterequest -n matrix
kubectl get challenges -n matrix

# Debug failed certificate issuance
kubectl describe challenge <challenge-name> -n matrix
kubectl logs -n cert-manager deployment/cert-manager -f

# Inspect the issued certificate
kubectl get secret <cert-secret-name> -n matrix -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout

SOPS Secret Editing

SOPS transparently encrypts/decrypts secrets using the age key specified in .sops.yaml. The environment variable SOPS_AGE_KEY_FILE must point to your age private key.

# Edit an encrypted secret (decrypted for editing, re-encrypted on save)
sops apps/production/custom-configs/mas-secret.yaml
sops apps/authentik/authentik-secret.yaml

# Create a new secret file
sops -i --encrypted-regex '^(data|stringData)$' --input-type yaml --output-type yaml new-secret.yaml

# Decrypt to view
sops -d apps/production/custom-configs/mas-secret.yaml

Ensure ~/.age/keys.txt exists and contains your age private key. See .devcontainer/devcontainer.json for setup details.

Helm Chart Inspection

# List installed charts
helm list -n matrix
helm list -n authentik

# View rendered chart values
helm get values matrix-stack -n matrix
helm get manifest matrix-stack -n matrix | less

Useful kubectl Shortcuts

# Port-forward to access services locally
kubectl port-forward -n matrix svc/synapse 8008:8008

# Execute command inside pod (for debugging)
kubectl exec -it <pod-name> -n matrix -- bash

# Stream logs from multiple pods
kubectl logs -n matrix -l app=synapse -f

# Bootstrap a service/bot account via MAS (no registration_shared_secret in this stack)
kubectl exec -it -n matrix deploy/matrix-stack-matrix-authentication-service -- \
  mas-cli manage register-user <name> --yes
kubectl exec -it -n matrix deploy/matrix-stack-matrix-authentication-service -- \
  mas-cli manage issue-compatibility-token <name>

Architecture & Key Concepts

FluxCD Reconciliation Flow

1. Flux watches clusters/matrix/ for a FluxRepository resource pointing to this Git repo
2. Kustomization stages pull in configurations in order:
   • flux-system (FluxCD itself)
   • infra-apps (Namespaces, RBAC, Cert-Manager, HelmRepository sources)
   • production-apps (Main ESS deployment and related services)
3. HelmReleases specify which charts to install and what values to use
4. ConfigMaps/Secrets provide values from files in the repo (e.g., custom Synapse config)
5. Flux auto-reconciles every 10 minutes, or immediately if Git changes are detected

Element Server Suite (ESS) Chart Constraints

The ESS Helm chart (v26.4.0) has strict validation and specific quirks:

• No config: blocks for core components — use ConfigMap overrides instead
• serverName must be at root level, not nested under synapse
• TLS in Ingress blocks is forbidden — use certManager: true at root to auto-manage certificates
• camelCase for component names: elementWeb, synapseAdmin, matrixAuthenticationService, etc.
• OCI HelmRepository only — the chart is distributed via oci://ghcr.io/element-hq/ess-helm, not HTTP
• Values must pass JSON schema validation — invalid configs will cause reconciliation failures with cryptic schema errors

NetworkPolicy Convention

Default-deny-with-explicit-allow across matrix and authentik namespaces (apps/production/networkpolicy.yaml, apps/authentik/networkpolicy.yaml). Every new pod needs its own explicit ingress-allow rule; NetworkPolicy matches on named container ports, not Service ports — a frequent source of live incidents when a new component is added (wrong port number/name silently blocks all traffic to it).

Known Issues & Workarounds

Issue: Let's Encrypt ACME Race Condition (Error 403 Order's status is processing)

• Symptom: Certificate provisioning hangs when elementWeb and wellKnownDelegation are both enabled on the same domain
• Cause: Both request certificates for the same domain simultaneously; Let's Encrypt rejects concurrent requests
• Fix: Set wellKnownDelegation: enabled: false and serve .well-known/matrix/server via a separate Ingress route or static file

Issue: HelmChart not ready / stat no such file or directory

• Cause: Attempting to use a GitRepository source for the ESS chart (it has sub-charts that don't render correctly)
• Fix: Use the OCI HelmRepository source (oci://ghcr.io/element-hq/ess-helm) instead

Issue: Certificate validation failures (No resources found)

• Cause: Manual Kustomize patches conflict with the Helm chart's built-in certificate management
• Fix: Remove manual patches; rely on certManager: true at the root level of HelmRelease values

Issue: Synapse module can't use asyncio

• Cause: Synapse runs on Twisted's reactor, not a running asyncio event loop — asyncio.open_connection/asyncio.wait_for inside a Synapse module (e.g. clamav_spam_checker.py) fail immediately with RuntimeError: no running event loop, and can silently trigger a fail-open path instead of an obvious crash
• Fix: use twisted.internet.reactor/HostnameEndpoint/connectProtocol + a custom Protocol subclass; Twisted Deferreds are natively awaitable from async def inside Synapse. Standalone processes outside Synapse (e.g. clamav-http-scanner.py) don't have this constraint and can use plain sockets/asyncio.

SOPS Encryption & Key Management

• .sops.yaml defines encryption rules (currently using age keys)
• Secrets matching the regex in .sops.yaml are automatically encrypted when committed
• The age private key (~/.age/keys.txt) must be available in your environment for decryption
• In the cluster, Flux decrypts secrets "on the fly" using a secret stored in flux-system namespace

To rotate SOPS keys:

# Regenerate and re-encrypt all secrets
sops updatekeys -y apps/

Development Workflow

Before Making Changes

1. Understand dependencies — check kustomization.yaml files to see the order of resource creation
2. Verify chart schema — review ESS chart documentation for constraints on the version being used
3. Test locally if possible — use kubectl port-forwards to verify connectivity before pushing changes

Making Changes

1. Edit ConfigMap files directly — for non-secret customizations (Synapse config, Element Web themes, etc.)
   • Changes are auto-tracked by git hooks installed via ./scripts/install-hooks.sh
2. Edit secrets with SOPS — sops transparently decrypts/re-encrypts on save
3. Update HelmRelease values — modify the values section in element-server-suite.yaml or reference ConfigMap sources

After Committing

1. Flux auto-detects changes within ~1 minute (or manually trigger with flux reconcile kustomization production-apps)
2. Monitor reconciliation — watch pod logs and Flux status for errors
3. Test functionality — verify services are accessible and functioning as expected

Git Hooks

After cloning, run:

./scripts/install-hooks.sh

This installs hooks that automatically commit ConfigMap changes to .gitignore-like tracking. See docs/ops-configmap-sync.md for details.

Environment Setup

Local Machine Prerequisites

• kubectl — cluster communication
• flux — GitOps CLI
• helm — chart inspection & debugging
• sops & age — secret management
• git — version control
• age key file at ~/.age/keys.txt (request from team)
• kubeconfig at ~/.kube/config (request from team)

DevContainer (Recommended)

The .devcontainer/ configuration provides a pre-configured environment:

# In VS Code: "Reopen in Container"
# Or manually:
docker build -t ess-devcontainer .devcontainer
docker run -it --rm \
  -v ~/.kube:/home/vscode/.kube \
  -v ~/.age:/home/vscode/.age \
  -v ~/.ssh:/home/vscode/.ssh \
  -v /var/run/docker.sock:/var/run/docker.sock \
  ess-devcontainer

DevContainer includes:

• All required CLI tools (kubectl, flux, helm, sops, age, git, docker)
• VS Code extensions for YAML, Kubernetes, Helm
• Proper environment variables (KUBECONFIG, SOPS_AGE_KEY_FILE)
• Git hooks pre-installed

Troubleshooting Checklist

• Pod not starting? → kubectl describe pod <name> -n matrix (check events)
• Image pull failures? → Check HelmRelease status: kubectl get helmrelease -n matrix
• Secret not found? → Verify SOPS decryption: sops -d <secret.sops.yaml> (must output valid YAML)
• Certificate stuck? → kubectl describe certificate <name> -n matrix (check for ACME errors)
• Config validation error? → Inspect HelmRelease status: kubectl describe helmrelease <name> -n matrix (JSON schema error message)
• Cluster unreachable? → Verify kubeconfig: kubectl get nodes (must connect to K3s)
• NetworkPolicy blocking a new pod? → Check it matches on container port name, not Service port

Resources & References

• README.md — High-level overview and architecture
• docs/TASKS.md — Task backlog, milestones, and priority list (open backlog lives in git.lab issues; the closed Gitea issues only point there)
• docs/deployment-guides/ — Detailed setup guides for specific components (01-07)
• docs/ops-configmap-sync.md — Git hook configuration and auto-sync behavior
• ESS Chart Docs — https://github.com/element-hq/ess-helm (official Helm chart repository)
• FluxCD Docs — https://fluxcd.io/docs/ (GitOps reconciliation & Kustomization)
• Matrix Spec — https://spec.matrix.org/ (Matrix protocol specification)

---

name: karpathy-guidelines description: Behavioral guidelines to reduce common LLM coding mistakes. Use when writing, reviewing, or refactoring code to avoid overcomplication, make surgical changes, surface assumptions, and define verifiable success criteria. license: MIT

Karpathy Guidelines

Behavioral guidelines to reduce common LLM coding mistakes, derived from Andrej Karpathy's observations on LLM coding pitfalls.

Tradeoff: These guidelines bias toward caution over speed. For trivial tasks, use judgment.

1. Think Before Coding

Don't assume. Don't hide confusion. Surface tradeoffs.

Before implementing:

• State your assumptions explicitly. If uncertain, ask.
• If multiple interpretations exist, present them - don't pick silently.
• If a simpler approach exists, say so. Push back when warranted.
• If something is unclear, stop. Name what's confusing. Ask.

2. Simplicity First

Minimum code that solves the problem. Nothing speculative.

• No features beyond what was asked.
• No abstractions for single-use code.
• No "flexibility" or "configurability" that wasn't requested.
• No error handling for impossible scenarios.
• If you write 200 lines and it could be 50, rewrite it.

Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify.

3. Surgical Changes

Touch only what you must. Clean up only your own mess.

When editing existing code:

• Don't "improve" adjacent code, comments, or formatting.
• Don't refactor things that aren't broken.
• Match existing style, even if you'd do it differently.
• If you notice unrelated dead code, mention it - don't delete it.

When your changes create orphans:

• Remove imports/variables/functions that YOUR changes made unused.
• Don't remove pre-existing dead code unless asked.

The test: Every changed line should trace directly to the user's request.

4. Goal-Driven Execution

Define success criteria. Loop until verified.

Transform tasks into verifiable goals:

• "Add validation" → "Write tests for invalid inputs, then make them pass"
• "Fix the bug" → "Write a test that reproduces it, then make it pass"
• "Refactor X" → "Ensure tests pass before and after"

For multi-step tasks, state a brief plan:

1. [Step] → verify: [check]
2. [Step] → verify: [check]
3. [Step] → verify: [check]

Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification.