feat(monitoring): scrape alertmanager, alert on failed delivery; fix stale README

Alertmanager was configured as an alerting target but never scraped, so its own
metrics were absent: a silently breaking alert chain could not report itself —
the same blind spot as a missing series, now at the end of the chain. Adds the
operating_alertmanager scrape job and AlertDeliveryFailing on
alertmanager_notifications_failed_total.

The README still claimed alert delivery was deliberately muted via a
room=security null receiver. That route is gone; alertmanager.yml routes
everything to the matrix receiver, so the backup alerts added yesterday do get
delivered. Documentation asserting the opposite is dangerous in both directions,
so it now states the current wiring and keeps the alert-storm history as
background.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
sorb
2026-08-14 12:00:00 +00:00
co-authored by Claude Opus 4.8
parent 4cb9bfb2bc
commit e9c13dcf22
3 changed files with 39 additions and 23 deletions
+12
View File
@@ -147,3 +147,15 @@ groups:
severity: critical
annotations:
summary: "Ein Backup-/Probe-CronJob fehlt im Cluster — Sicherung oder Wiederherstellungs-Nachweis laeuft nicht mehr"
# Die Alarmkette meldet ihr eigenes Reissen. Ironie inklusive: schlaegt die
# Zustellung komplett fehl, kommt auch dieser Alarm nicht an - er ist dann
# aber in Prometheus/Grafana sichtbar, statt dass gar nichts existiert.
# Teilausfaelle (ein Receiver von mehreren, Rate-Limit 429) meldet er sauber.
- alert: AlertDeliveryFailing
expr: increase(alertmanager_notifications_failed_total[15m]) > 0
labels:
severity: critical
annotations:
summary: "Alertmanager konnte Benachrichtigungen nicht zustellen ({{ $labels.integration }}) — Alarme laufen ins Leere"