Thore Cimbal
9a3b1d1dd5
Restarting k3s took kube-state-metrics and alloy's log tailers down with it, so kube_pod_container_info went empty in prometheus. The derivation asked, got a clean response with zero rows, and counted it as success. Cluster targets went from 39 to none, the total from 54 to 12, and the scan loop deleted every report whose target had vanished. Coverage then read 1.0. None of the four rules fired, and each for a defensible reason: the set was not empty because the operating host still answered, and every timestamp was fresh because an empty success updates it. The gap sat exactly between them. A source that has delivered before and now delivers nothing is treated as a failure: its previous targets are kept, its timestamp ages, and the stale rule takes over. A fifth rule watches the total for a drop of more than 40 percent, and it deliberately also fires on a deliberate shrink — losing 40 percent of the checked estate is worth a line either way. The six rancher decisions are gone too. They described versions that no longer run: the k3s patch took all fifteen of their criticals with it.
threadnet-operating
Der Betriebs-/Monitoring-Stack für den Operating-Host CFGMON: Prometheus,
Loki, Grafana, Alloy, Alertmanager und der CVE-Exporter — vollständig als Code,
ein docker compose up -d stellt ihn auf einem frischen Host wieder her.
→ monitoring/README.md ist die eigentliche
Betriebsanleitung (Deployment, Config-Fallen, Alerting, CVE-Pipeline).
Wo was liegt
| Pfad | Inhalt |
|---|---|
monitoring/ |
der Stack: Compose, Prometheus, Loki, Grafana, Alertmanager, Alloy |
monitoring/cve/ |
CVE-Exporter (Trivy-Scan → Prometheus-Metriken), ADR-0003 |
monitoring/grafana/ |
Datasources und Dashboards als Code |
Offene Punkte
Kein Backlog in diesem Repo. Offene Punkte sind Issues im
management-Projekt
(ADR-0005) —
sie betreffen meist mehrere Hosts, eine Liste je Repo würde auseinanderlaufen.
Für diesen Stack einschlägig sind unter anderem
#8 Remote-Write und Loki ohne Auth,
#9 Grafana-Credentials und
#10 Gitea-Backups off-host;
Bestand und Historie zum Host stehen in
hosts/cfgmon.md.
Kanonisch ist git.lab (ADR-0001,
ADR-0002).
Von außerhalb des Labs ist derselbe Stand über den Push-Mirror
sorb/management lesbar — dorthin
aber nie pushen, der Mirror überschreibt.