# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Overview

This is a **GitOps-based Kubernetes deployment** of **Element Server Suite (ESS Community v26.4.0)**, a complete Matrix homeserver stack. The repository contains Infrastructure-as-Code using **FluxCD** for GitOps synchronization, with encryption (SOPS/age), service mesh (Traefik), certificate management (Cert-Manager), and auxiliary services like Authentik, TURN/coturn, Draupnir (moderation), ClamAV (content scanning), and Grafana monitoring.

**Key Stack Components:**
- **K3s**: Lightweight Kubernetes distribution running on Hetzner Cloud
- **FluxCD**: GitOps controller that watches this repository and auto-syncs changes
- **ESS (Matrix Umbrella Chart v26.4.0)**: Synapse, MAS (Matrix Authentication Service), ElementWeb, MatrixRTC
- **Authentik**: OIDC-based identity provider for centralized authentication, deployed both via HelmRelease and declarative Blueprints (`apps/authentik/authentik-blueprints.yaml`) for flows/OIDC-provider config that would otherwise only exist as manual admin-UI clicks
- **Traefik**: Ingress controller (built into K3s) for routing HTTP/HTTPS traffic
- **Cert-Manager**: Automatic TLS certificate provisioning from Let's Encrypt
- **SOPS + age**: Transparent encryption/decryption of secrets in Git
- **Monitoring**: Grafana Alloy (agent), Prometheus (metrics), Loki (logs)
- **coturn**: TURN/STUN server for WebRTC audio/video calls, with monthly automated shared-secret rotation via CronJob + PR workflow
- **Draupnir**: Matrix moderation bot (community successor to Mjolnir), ban lists/policy rooms
- **ClamAV**: Content scanning — a Synapse module for unencrypted-room uploads, plus a standalone `clamav-http-scanner` service that a patched Element Web client (ThreadNet-Web) calls both on send and on receive, extending coverage to encrypted rooms/DMs
- **NetworkPolicies**: default-deny-with-explicit-allow across `matrix` and `authentik` namespaces
- **`host-config/`**: the one part of this repo that is deliberately **not** managed by Flux/GitOps — see "Host-Level (non-GitOps) Changes" below

## Repo Topology (since 2026-07-31)

Canonical repo is **`git.lab/axion1337.chat/axion1337.chat-gitops`** (homelab GitLab,
resolvable only inside the lab) — all pushes go there; a push-mirror updates the Gitea
copy on `rohana.axion1337.de`, which remains the **Flux source** (the cluster pulls from
Gitea; the mirror delivers). **Never push directly to Gitea** for this repo — the mirror
force-overwrites divergent state. Issues/wiki/releases stay on Gitea. The same rule
applies to ThreadNet-Web, threadnet-call, thread-net-git and threadnet-operating; only
the `Backlogs` repo (and this repo's wiki) are still direct-to-Gitea.

## Repository Structure

```
gitops/
├── clusters/matrix/          # Flux GitRepository definition; entry point for reconciliation
├── apps/
│   ├── base/
│   │   ├── infra/            # Core infrastructure (Cert-Manager, Namespaces, etc.)
│   │   └── matrix/           # HelmRepository definition for ESS OCI chart
│   ├── production/           # Main ESS deployment
│   │   ├── element-server-suite.yaml     # HelmRelease (ESS chart v26.4.0)
│   │   ├── custom-configs/               # Overrides & custom configurations
│   │   │   ├── synapse-values.yaml       # Synapse customizations (ConfigMap)
│   │   │   ├── element-values.yaml       # ElementWeb customizations (ConfigMap)
│   │   │   └── mas-secret.yaml           # MAS secrets (encrypted with SOPS)
│   │   ├── cert-issuer.yaml              # Let's Encrypt ClusterIssuer
│   │   ├── apex-ingress.yaml              # Apex-domain IngressRoutes (Element Web, /_scan, etc.)
│   │   ├── matrix-postgres-auth.yaml     # PostgreSQL credentials
│   │   ├── coturn.yaml / coturn-secret.yaml / synapse-turn-secret.yaml
│   │   ├── turn-secret-rotation.yaml     # Monthly CronJob, rotates coturn shared secret via PR
│   │   ├── draupnir.yaml / draupnir-pvc.yaml / draupnir-secret.yaml
│   │   ├── clamav.yaml / clamav-pvc.yaml / clamav_spam_checker.py   # Synapse-side scan module
│   │   ├── clamav-http-scanner.py / -Dockerfile / .yaml             # Client-side scan service
│   │   ├── synapse-backup.yaml / synapse-backup-secret.yaml
│   │   └── networkpolicy.yaml            # Default-deny + explicit allow rules
│   ├── authentik/            # Identity Provider (separate namespace)
│   │   ├── authentik.yaml                # HelmRelease
│   │   ├── authentik-blueprints.yaml     # Flows/OIDC-provider as declarative code
│   │   ├── helm-repo.yaml                # HelmRepository source
│   │   ├── ingress.yaml                  # Ingress route
│   │   ├── networkpolicy.yaml
│   │   └── authentik-secret.yaml         # Secrets (admin password, OIDC client secret, etc.)
│   └── monitoring/           # Observability (Alloy, kube-state-metrics, node-exporter)
│       ├── alloy-config.yaml             # Grafana Alloy configuration
│       └── kube-state-metrics.yaml       # K8s metrics exporter
├── host-config/               # Host-level (non-GitOps) config, see below
│   └── maintenance-notify/    # systemd timer: pre-update mail/Matrix notifications (Issue #24)
├── .sops.yaml                # SOPS encryption rules (age key definition)
├── scripts/
│   ├── install-hooks.sh      # Installs git hooks for ConfigMap auto-tracking
│   └── hooks/                # Git hooks (pre-commit, post-commit, etc.)
└── docs/
    ├── README.md             # Main deployment guide
    ├── TASKS.md               # Task list & milestones (backlog itself lives in Gitea issues)
    ├── install.md             # Installation instructions
    ├── ops-configmap-sync.md # ConfigMap syncing with git hooks
    └── deployment-guides/    # Detailed guides for specific components (01-07)
```

## Host-Level (non-GitOps) Changes

Almost everything in this repo is reconciled by Flux. `host-config/` is the deliberate
exception: it holds scripts/systemd units meant to run **on the bare Hetzner host itself**
(not as a Kubernetes pod), for things Flux structurally can't reach — e.g. host package
management. There is no SOPS-on-host or Ansible-equivalent mechanism yet; deployment to the
host is manual (`scp`/SSH), and instance-specific values live in a config file on the host
(`/etc/<name>/config`), not hardcoded in the versioned script, so the pattern is reusable
across forks/other communities running this same stack. See
`docs/deployment-guides/07-host-maintenance-notifications.md` for the first (and so far only)
example of this pattern.

## Common Development Commands

### Flux / GitOps Synchronization

```bash
# Force immediate reconciliation (don't wait for 10-min auto-sync)
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization production-apps --with-source

# Check reconciliation status
flux get kustomizations -A
flux get helmreleases -A

# View Flux logs
kubectl logs -n flux-system deployment/source-controller -f
kubectl logs -n flux-system deployment/helm-controller -f
```

### Kubernetes Cluster Status

```bash
# Check pod health in Matrix namespace
kubectl get pods -n matrix
kubectl get pods -n authentik
kubectl get pods -n monitoring

# Detailed pod inspection
kubectl describe pod <pod-name> -n matrix
kubectl logs <pod-name> -n matrix -f

# Check all services and ingresses
kubectl get svc -n matrix
kubectl get ingress -n matrix
```

### Certificate Management (Let's Encrypt / Cert-Manager)

```bash
# View certificate status
kubectl get certificate -n matrix
kubectl get certificaterequest -n matrix
kubectl get challenges -n matrix

# Debug failed certificate issuance
kubectl describe challenge <challenge-name> -n matrix
kubectl logs -n cert-manager deployment/cert-manager -f

# Inspect the issued certificate
kubectl get secret <cert-secret-name> -n matrix -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout
```

### SOPS Secret Editing

SOPS transparently encrypts/decrypts secrets using the `age` key specified in `.sops.yaml`. The environment variable `SOPS_AGE_KEY_FILE` must point to your age private key.

```bash
# Edit an encrypted secret (decrypted for editing, re-encrypted on save)
sops apps/production/custom-configs/mas-secret.yaml
sops apps/authentik/authentik-secret.yaml

# Create a new secret file
sops -i --encrypted-regex '^(data|stringData)$' --input-type yaml --output-type yaml new-secret.yaml

# Decrypt to view
sops -d apps/production/custom-configs/mas-secret.yaml
```

Ensure `~/.age/keys.txt` exists and contains your age private key. See `.devcontainer/devcontainer.json` for setup details.

### Helm Chart Inspection

```bash
# List installed charts
helm list -n matrix
helm list -n authentik

# View rendered chart values
helm get values matrix-stack -n matrix
helm get manifest matrix-stack -n matrix | less
```

### Useful kubectl Shortcuts

```bash
# Port-forward to access services locally
kubectl port-forward -n matrix svc/synapse 8008:8008

# Execute command inside pod (for debugging)
kubectl exec -it <pod-name> -n matrix -- bash

# Stream logs from multiple pods
kubectl logs -n matrix -l app=synapse -f

# Bootstrap a service/bot account via MAS (no registration_shared_secret in this stack)
kubectl exec -it -n matrix deploy/matrix-stack-matrix-authentication-service -- \
  mas-cli manage register-user <name> --yes
kubectl exec -it -n matrix deploy/matrix-stack-matrix-authentication-service -- \
  mas-cli manage issue-compatibility-token <name>
```

## Architecture & Key Concepts

### FluxCD Reconciliation Flow

1. **Flux watches** `clusters/matrix/` for a FluxRepository resource pointing to this Git repo
2. **Kustomization stages** pull in configurations in order:
   - `flux-system` (FluxCD itself)
   - `infra-apps` (Namespaces, RBAC, Cert-Manager, HelmRepository sources)
   - `production-apps` (Main ESS deployment and related services)
3. **HelmReleases** specify which charts to install and what values to use
4. **ConfigMaps/Secrets** provide values from files in the repo (e.g., custom Synapse config)
5. **Flux auto-reconciles** every 10 minutes, or immediately if Git changes are detected

### Element Server Suite (ESS) Chart Constraints

The ESS Helm chart (v26.4.0) has strict validation and specific quirks:

- **No `config:` blocks for core components** — use ConfigMap overrides instead
- **`serverName` must be at root level**, not nested under `synapse`
- **TLS in Ingress blocks is forbidden** — use `certManager: true` at root to auto-manage certificates
- **`camelCase` for component names**: `elementWeb`, `synapseAdmin`, `matrixAuthenticationService`, etc.
- **OCI HelmRepository only** — the chart is distributed via `oci://ghcr.io/element-hq/ess-helm`, not HTTP
- **Values must pass JSON schema validation** — invalid configs will cause reconciliation failures with cryptic schema errors

### NetworkPolicy Convention

Default-deny-with-explicit-allow across `matrix` and `authentik` namespaces
(`apps/production/networkpolicy.yaml`, `apps/authentik/networkpolicy.yaml`). Every new pod
needs its own explicit ingress-allow rule; NetworkPolicy matches on named **container ports**,
not Service ports — a frequent source of live incidents when a new component is added (wrong
port number/name silently blocks all traffic to it).

### Known Issues & Workarounds

**Issue: Let's Encrypt ACME Race Condition (Error 403 Order's status is processing)**
- Symptom: Certificate provisioning hangs when `elementWeb` and `wellKnownDelegation` are both enabled on the same domain
- Cause: Both request certificates for the same domain simultaneously; Let's Encrypt rejects concurrent requests
- Fix: Set `wellKnownDelegation: enabled: false` and serve `.well-known/matrix/server` via a separate Ingress route or static file

**Issue: HelmChart not ready / stat no such file or directory**
- Cause: Attempting to use a GitRepository source for the ESS chart (it has sub-charts that don't render correctly)
- Fix: Use the OCI HelmRepository source (`oci://ghcr.io/element-hq/ess-helm`) instead

**Issue: Certificate validation failures (No resources found)**
- Cause: Manual Kustomize patches conflict with the Helm chart's built-in certificate management
- Fix: Remove manual patches; rely on `certManager: true` at the root level of HelmRelease values

**Issue: Synapse module can't use asyncio**
- Cause: Synapse runs on Twisted's reactor, not a running asyncio event loop — `asyncio.open_connection`/`asyncio.wait_for` inside a Synapse module (e.g. `clamav_spam_checker.py`) fail immediately with `RuntimeError: no running event loop`, and can silently trigger a fail-open path instead of an obvious crash
- Fix: use `twisted.internet.reactor`/`HostnameEndpoint`/`connectProtocol` + a custom `Protocol` subclass; Twisted `Deferred`s are natively awaitable from `async def` inside Synapse. Standalone processes outside Synapse (e.g. `clamav-http-scanner.py`) don't have this constraint and can use plain sockets/asyncio.

### SOPS Encryption & Key Management

- `.sops.yaml` defines encryption rules (currently using `age` keys)
- Secrets matching the regex in `.sops.yaml` are automatically encrypted when committed
- The age private key (`~/.age/keys.txt`) must be available in your environment for decryption
- In the cluster, Flux decrypts secrets "on the fly" using a secret stored in `flux-system` namespace

To rotate SOPS keys:
```bash
# Regenerate and re-encrypt all secrets
sops updatekeys -y apps/
```

## Development Workflow

### Before Making Changes

1. **Understand dependencies** — check `kustomization.yaml` files to see the order of resource creation
2. **Verify chart schema** — review ESS chart documentation for constraints on the version being used
3. **Test locally if possible** — use `kubectl` port-forwards to verify connectivity before pushing changes

### Making Changes

1. **Edit ConfigMap files directly** — for non-secret customizations (Synapse config, Element Web themes, etc.)
   - Changes are auto-tracked by git hooks installed via `./scripts/install-hooks.sh`
2. **Edit secrets with SOPS** — `sops` transparently decrypts/re-encrypts on save
3. **Update HelmRelease values** — modify the `values` section in `element-server-suite.yaml` or reference ConfigMap sources

### After Committing

1. **Flux auto-detects changes** within ~1 minute (or manually trigger with `flux reconcile kustomization production-apps`)
2. **Monitor reconciliation** — watch pod logs and Flux status for errors
3. **Test functionality** — verify services are accessible and functioning as expected

### Git Hooks

After cloning, run:
```bash
./scripts/install-hooks.sh
```

This installs hooks that automatically commit ConfigMap changes to `.gitignore`-like tracking. See `docs/ops-configmap-sync.md` for details.

## Environment Setup

### Local Machine Prerequisites

- `kubectl` — cluster communication
- `flux` — GitOps CLI
- `helm` — chart inspection & debugging
- `sops` & `age` — secret management
- `git` — version control
- age key file at `~/.age/keys.txt` (request from team)
- kubeconfig at `~/.kube/config` (request from team)

### DevContainer (Recommended)

The `.devcontainer/` configuration provides a pre-configured environment:

```bash
# In VS Code: "Reopen in Container"
# Or manually:
docker build -t ess-devcontainer .devcontainer
docker run -it --rm \
  -v ~/.kube:/home/vscode/.kube \
  -v ~/.age:/home/vscode/.age \
  -v ~/.ssh:/home/vscode/.ssh \
  -v /var/run/docker.sock:/var/run/docker.sock \
  ess-devcontainer
```

DevContainer includes:
- All required CLI tools (kubectl, flux, helm, sops, age, git, docker)
- VS Code extensions for YAML, Kubernetes, Helm
- Proper environment variables (`KUBECONFIG`, `SOPS_AGE_KEY_FILE`)
- Git hooks pre-installed

## Troubleshooting Checklist

- **Pod not starting?** → `kubectl describe pod <name> -n matrix` (check events)
- **Image pull failures?** → Check HelmRelease status: `kubectl get helmrelease -n matrix`
- **Secret not found?** → Verify SOPS decryption: `sops -d <secret.sops.yaml>` (must output valid YAML)
- **Certificate stuck?** → `kubectl describe certificate <name> -n matrix` (check for ACME errors)
- **Config validation error?** → Inspect HelmRelease status: `kubectl describe helmrelease <name> -n matrix` (JSON schema error message)
- **Cluster unreachable?** → Verify kubeconfig: `kubectl get nodes` (must connect to K3s)
- **NetworkPolicy blocking a new pod?** → Check it matches on container port name, not Service port

## Resources & References

- **README.md** — High-level overview and architecture
- **docs/TASKS.md** — Task backlog, milestones, and priority list (open backlog lives in [Gitea issues](https://rohana.axion1337.de/sorb/axion1337.chat-gitops/issues))
- **docs/deployment-guides/** — Detailed setup guides for specific components (01-07)
- **docs/ops-configmap-sync.md** — Git hook configuration and auto-sync behavior
- **ESS Chart Docs** — `https://github.com/element-hq/ess-helm` (official Helm chart repository)
- **FluxCD Docs** — `https://fluxcd.io/docs/` (GitOps reconciliation & Kustomization)
- **Matrix Spec** — `https://spec.matrix.org/` (Matrix protocol specification)