axion1337.chat-gitops/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Overview

This is a GitOps-based Kubernetes deployment of Element Server Suite (ESS Community v26.4.0), a complete Matrix homeserver stack. The repository contains Infrastructure-as-Code using FluxCD for GitOps synchronization, with encryption (SOPS/age), service mesh (Traefik), certificate management (Cert-Manager), and auxiliary services like Authentik, TURN/coturn, Draupnir (moderation), ClamAV (content scanning), and Grafana monitoring.

Key Stack Components:

• K3s: Lightweight Kubernetes distribution running on Hetzner Cloud
• FluxCD: GitOps controller that watches this repository and auto-syncs changes
• ESS (Matrix Umbrella Chart v26.4.0): Synapse, MAS (Matrix Authentication Service), ElementWeb, MatrixRTC
• Authentik: OIDC-based identity provider for centralized authentication, deployed both via HelmRelease and declarative Blueprints (apps/authentik/authentik-blueprints.yaml) for flows/OIDC-provider config that would otherwise only exist as manual admin-UI clicks
• Traefik: Ingress controller (built into K3s) for routing HTTP/HTTPS traffic
• Cert-Manager: Automatic TLS certificate provisioning from Let's Encrypt
• SOPS + age: Transparent encryption/decryption of secrets in Git
• Monitoring: Grafana Alloy (agent), Prometheus (metrics), Loki (logs)
• coturn: TURN/STUN server for WebRTC audio/video calls, with monthly automated shared-secret rotation via CronJob + PR workflow
• Draupnir: Matrix moderation bot (community successor to Mjolnir), ban lists/policy rooms
• ClamAV: Content scanning — a Synapse module for unencrypted-room uploads, plus a standalone clamav-http-scanner service that a patched Element Web client (ThreadNet-Web) calls both on send and on receive, extending coverage to encrypted rooms/DMs
• NetworkPolicies: default-deny-with-explicit-allow across matrix and authentik namespaces
• host-config/: the one part of this repo that is deliberately not managed by Flux/GitOps — see "Host-Level (non-GitOps) Changes" below

Repository Structure

gitops/
├── clusters/matrix/          # Flux GitRepository definition; entry point for reconciliation
├── apps/
│   ├── base/
│   │   ├── infra/            # Core infrastructure (Cert-Manager, Namespaces, etc.)
│   │   └── matrix/           # HelmRepository definition for ESS OCI chart
│   ├── production/           # Main ESS deployment
│   │   ├── element-server-suite.yaml     # HelmRelease (ESS chart v26.4.0)
│   │   ├── custom-configs/               # Overrides & custom configurations
│   │   │   ├── synapse-values.yaml       # Synapse customizations (ConfigMap)
│   │   │   ├── element-values.yaml       # ElementWeb customizations (ConfigMap)
│   │   │   └── mas-secret.yaml           # MAS secrets (encrypted with SOPS)
│   │   ├── cert-issuer.yaml              # Let's Encrypt ClusterIssuer
│   │   ├── apex-ingress.yaml              # Apex-domain IngressRoutes (Element Web, /_scan, etc.)
│   │   ├── matrix-postgres-auth.yaml     # PostgreSQL credentials
│   │   ├── coturn.yaml / coturn-secret.yaml / synapse-turn-secret.yaml
│   │   ├── turn-secret-rotation.yaml     # Monthly CronJob, rotates coturn shared secret via PR
│   │   ├── draupnir.yaml / draupnir-pvc.yaml / draupnir-secret.yaml
│   │   ├── clamav.yaml / clamav-pvc.yaml / clamav_spam_checker.py   # Synapse-side scan module
│   │   ├── clamav-http-scanner.py / -Dockerfile / .yaml             # Client-side scan service
│   │   ├── synapse-backup.yaml / synapse-backup-secret.yaml
│   │   └── networkpolicy.yaml            # Default-deny + explicit allow rules
│   ├── authentik/            # Identity Provider (separate namespace)
│   │   ├── authentik.yaml                # HelmRelease
│   │   ├── authentik-blueprints.yaml     # Flows/OIDC-provider as declarative code
│   │   ├── helm-repo.yaml                # HelmRepository source
│   │   ├── ingress.yaml                  # Ingress route
│   │   ├── networkpolicy.yaml
│   │   └── authentik-secret.yaml         # Secrets (admin password, OIDC client secret, etc.)
│   └── monitoring/           # Observability (Alloy, kube-state-metrics, node-exporter)
│       ├── alloy-config.yaml             # Grafana Alloy configuration
│       └── kube-state-metrics.yaml       # K8s metrics exporter
├── host-config/               # Host-level (non-GitOps) config, see below
│   └── maintenance-notify/    # systemd timer: pre-update mail/Matrix notifications (Issue #24)
├── .sops.yaml                # SOPS encryption rules (age key definition)
├── scripts/
│   ├── install-hooks.sh      # Installs git hooks for ConfigMap auto-tracking
│   └── hooks/                # Git hooks (pre-commit, post-commit, etc.)
└── docs/
    ├── README.md             # Main deployment guide
    ├── TASKS.md               # Task list & milestones (backlog itself lives in Gitea issues)
    ├── install.md             # Installation instructions
    ├── ops-configmap-sync.md # ConfigMap syncing with git hooks
    └── deployment-guides/    # Detailed guides for specific components (01-07)

Host-Level (non-GitOps) Changes

Almost everything in this repo is reconciled by Flux. host-config/ is the deliberate exception: it holds scripts/systemd units meant to run on the bare Hetzner host itself (not as a Kubernetes pod), for things Flux structurally can't reach — e.g. host package management. There is no SOPS-on-host or Ansible-equivalent mechanism yet; deployment to the host is manual (scp/SSH), and instance-specific values live in a config file on the host (/etc/<name>/config), not hardcoded in the versioned script, so the pattern is reusable across forks/other communities running this same stack. See docs/deployment-guides/07-host-maintenance-notifications.md for the first (and so far only) example of this pattern.

Common Development Commands

Flux / GitOps Synchronization

# Force immediate reconciliation (don't wait for 10-min auto-sync)
flux reconcile kustomization flux-system --with-source
flux reconcile kustomization production-apps --with-source

# Check reconciliation status
flux get kustomizations -A
flux get helmreleases -A

# View Flux logs
kubectl logs -n flux-system deployment/source-controller -f
kubectl logs -n flux-system deployment/helm-controller -f

Kubernetes Cluster Status

# Check pod health in Matrix namespace
kubectl get pods -n matrix
kubectl get pods -n authentik
kubectl get pods -n monitoring

# Detailed pod inspection
kubectl describe pod <pod-name> -n matrix
kubectl logs <pod-name> -n matrix -f

# Check all services and ingresses
kubectl get svc -n matrix
kubectl get ingress -n matrix

Certificate Management (Let's Encrypt / Cert-Manager)

# View certificate status
kubectl get certificate -n matrix
kubectl get certificaterequest -n matrix
kubectl get challenges -n matrix

# Debug failed certificate issuance
kubectl describe challenge <challenge-name> -n matrix
kubectl logs -n cert-manager deployment/cert-manager -f

# Inspect the issued certificate
kubectl get secret <cert-secret-name> -n matrix -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout

SOPS Secret Editing

SOPS transparently encrypts/decrypts secrets using the age key specified in .sops.yaml. The environment variable SOPS_AGE_KEY_FILE must point to your age private key.

# Edit an encrypted secret (decrypted for editing, re-encrypted on save)
sops apps/production/custom-configs/mas-secret.yaml
sops apps/authentik/authentik-secret.yaml

# Create a new secret file
sops -i --encrypted-regex '^(data|stringData)$' --input-type yaml --output-type yaml new-secret.yaml

# Decrypt to view
sops -d apps/production/custom-configs/mas-secret.yaml

Ensure ~/.age/keys.txt exists and contains your age private key. See .devcontainer/devcontainer.json for setup details.

Helm Chart Inspection

# List installed charts
helm list -n matrix
helm list -n authentik

# View rendered chart values
helm get values matrix-stack -n matrix
helm get manifest matrix-stack -n matrix | less

Useful kubectl Shortcuts

# Port-forward to access services locally
kubectl port-forward -n matrix svc/synapse 8008:8008

# Execute command inside pod (for debugging)
kubectl exec -it <pod-name> -n matrix -- bash

# Stream logs from multiple pods
kubectl logs -n matrix -l app=synapse -f

# Bootstrap a service/bot account via MAS (no registration_shared_secret in this stack)
kubectl exec -it -n matrix deploy/matrix-stack-matrix-authentication-service -- \
  mas-cli manage register-user <name> --yes
kubectl exec -it -n matrix deploy/matrix-stack-matrix-authentication-service -- \
  mas-cli manage issue-compatibility-token <name>

Architecture & Key Concepts

FluxCD Reconciliation Flow

1. Flux watches clusters/matrix/ for a FluxRepository resource pointing to this Git repo
2. Kustomization stages pull in configurations in order:
   • flux-system (FluxCD itself)
   • infra-apps (Namespaces, RBAC, Cert-Manager, HelmRepository sources)
   • production-apps (Main ESS deployment and related services)
3. HelmReleases specify which charts to install and what values to use
4. ConfigMaps/Secrets provide values from files in the repo (e.g., custom Synapse config)
5. Flux auto-reconciles every 10 minutes, or immediately if Git changes are detected

Element Server Suite (ESS) Chart Constraints

The ESS Helm chart (v26.4.0) has strict validation and specific quirks:

• No config: blocks for core components — use ConfigMap overrides instead
• serverName must be at root level, not nested under synapse
• TLS in Ingress blocks is forbidden — use certManager: true at root to auto-manage certificates
• camelCase for component names: elementWeb, synapseAdmin, matrixAuthenticationService, etc.
• OCI HelmRepository only — the chart is distributed via oci://ghcr.io/element-hq/ess-helm, not HTTP
• Values must pass JSON schema validation — invalid configs will cause reconciliation failures with cryptic schema errors

NetworkPolicy Convention

Default-deny-with-explicit-allow across matrix and authentik namespaces (apps/production/networkpolicy.yaml, apps/authentik/networkpolicy.yaml). Every new pod needs its own explicit ingress-allow rule; NetworkPolicy matches on named container ports, not Service ports — a frequent source of live incidents when a new component is added (wrong port number/name silently blocks all traffic to it).

Known Issues & Workarounds

Issue: Let's Encrypt ACME Race Condition (Error 403 Order's status is processing)

• Symptom: Certificate provisioning hangs when elementWeb and wellKnownDelegation are both enabled on the same domain
• Cause: Both request certificates for the same domain simultaneously; Let's Encrypt rejects concurrent requests
• Fix: Set wellKnownDelegation: enabled: false and serve .well-known/matrix/server via a separate Ingress route or static file

Issue: HelmChart not ready / stat no such file or directory

• Cause: Attempting to use a GitRepository source for the ESS chart (it has sub-charts that don't render correctly)
• Fix: Use the OCI HelmRepository source (oci://ghcr.io/element-hq/ess-helm) instead

Issue: Certificate validation failures (No resources found)

• Cause: Manual Kustomize patches conflict with the Helm chart's built-in certificate management
• Fix: Remove manual patches; rely on certManager: true at the root level of HelmRelease values

Issue: Synapse module can't use asyncio

• Cause: Synapse runs on Twisted's reactor, not a running asyncio event loop — asyncio.open_connection/asyncio.wait_for inside a Synapse module (e.g. clamav_spam_checker.py) fail immediately with RuntimeError: no running event loop, and can silently trigger a fail-open path instead of an obvious crash
• Fix: use twisted.internet.reactor/HostnameEndpoint/connectProtocol + a custom Protocol subclass; Twisted Deferreds are natively awaitable from async def inside Synapse. Standalone processes outside Synapse (e.g. clamav-http-scanner.py) don't have this constraint and can use plain sockets/asyncio.

SOPS Encryption & Key Management

• .sops.yaml defines encryption rules (currently using age keys)
• Secrets matching the regex in .sops.yaml are automatically encrypted when committed
• The age private key (~/.age/keys.txt) must be available in your environment for decryption
• In the cluster, Flux decrypts secrets "on the fly" using a secret stored in flux-system namespace

To rotate SOPS keys:

# Regenerate and re-encrypt all secrets
sops updatekeys -y apps/

Development Workflow

Before Making Changes

1. Understand dependencies — check kustomization.yaml files to see the order of resource creation
2. Verify chart schema — review ESS chart documentation for constraints on the version being used
3. Test locally if possible — use kubectl port-forwards to verify connectivity before pushing changes

Making Changes

1. Edit ConfigMap files directly — for non-secret customizations (Synapse config, Element Web themes, etc.)
   • Changes are auto-tracked by git hooks installed via ./scripts/install-hooks.sh
2. Edit secrets with SOPS — sops transparently decrypts/re-encrypts on save
3. Update HelmRelease values — modify the values section in element-server-suite.yaml or reference ConfigMap sources

After Committing

1. Flux auto-detects changes within ~1 minute (or manually trigger with flux reconcile kustomization production-apps)
2. Monitor reconciliation — watch pod logs and Flux status for errors
3. Test functionality — verify services are accessible and functioning as expected

Git Hooks

After cloning, run:

./scripts/install-hooks.sh

This installs hooks that automatically commit ConfigMap changes to .gitignore-like tracking. See docs/ops-configmap-sync.md for details.

Environment Setup

Local Machine Prerequisites

• kubectl — cluster communication
• flux — GitOps CLI
• helm — chart inspection & debugging
• sops & age — secret management
• git — version control
• age key file at ~/.age/keys.txt (request from team)
• kubeconfig at ~/.kube/config (request from team)

DevContainer (Recommended)

The .devcontainer/ configuration provides a pre-configured environment:

# In VS Code: "Reopen in Container"
# Or manually:
docker build -t ess-devcontainer .devcontainer
docker run -it --rm \
  -v ~/.kube:/home/vscode/.kube \
  -v ~/.age:/home/vscode/.age \
  -v ~/.ssh:/home/vscode/.ssh \
  -v /var/run/docker.sock:/var/run/docker.sock \
  ess-devcontainer

DevContainer includes:

• All required CLI tools (kubectl, flux, helm, sops, age, git, docker)
• VS Code extensions for YAML, Kubernetes, Helm
• Proper environment variables (KUBECONFIG, SOPS_AGE_KEY_FILE)
• Git hooks pre-installed

Troubleshooting Checklist

• Pod not starting? → kubectl describe pod <name> -n matrix (check events)
• Image pull failures? → Check HelmRelease status: kubectl get helmrelease -n matrix
• Secret not found? → Verify SOPS decryption: sops -d <secret.sops.yaml> (must output valid YAML)
• Certificate stuck? → kubectl describe certificate <name> -n matrix (check for ACME errors)
• Config validation error? → Inspect HelmRelease status: kubectl describe helmrelease <name> -n matrix (JSON schema error message)
• Cluster unreachable? → Verify kubeconfig: kubectl get nodes (must connect to K3s)
• NetworkPolicy blocking a new pod? → Check it matches on container port name, not Service port

Resources & References

• README.md — High-level overview and architecture
• docs/TASKS.md — Task backlog, milestones, and priority list (open backlog lives in Gitea issues)
• docs/deployment-guides/ — Detailed setup guides for specific components (01-07)
• docs/ops-configmap-sync.md — Git hook configuration and auto-sync behavior
• ESS Chart Docs — https://github.com/element-hq/ess-helm (official Helm chart repository)
• FluxCD Docs — https://fluxcd.io/docs/ (GitOps reconciliation & Kustomization)
• Matrix Spec — https://spec.matrix.org/ (Matrix protocol specification)