feat(monitoring): scrape alertmanager, alert on failed delivery; fix stale README
Alertmanager was configured as an alerting target but never scraped, so its own metrics were absent: a silently breaking alert chain could not report itself — the same blind spot as a missing series, now at the end of the chain. Adds the operating_alertmanager scrape job and AlertDeliveryFailing on alertmanager_notifications_failed_total. The README still claimed alert delivery was deliberately muted via a room=security null receiver. That route is gone; alertmanager.yml routes everything to the matrix receiver, so the backup alerts added yesterday do get delivered. Documentation asserting the opposite is dangerous in both directions, so it now states the current wiring and keeps the alert-storm history as background. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -147,3 +147,15 @@ groups:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Ein Backup-/Probe-CronJob fehlt im Cluster — Sicherung oder Wiederherstellungs-Nachweis laeuft nicht mehr"
|
||||
|
||||
# Die Alarmkette meldet ihr eigenes Reissen. Ironie inklusive: schlaegt die
|
||||
# Zustellung komplett fehl, kommt auch dieser Alarm nicht an - er ist dann
|
||||
# aber in Prometheus/Grafana sichtbar, statt dass gar nichts existiert.
|
||||
# Teilausfaelle (ein Receiver von mehreren, Rate-Limit 429) meldet er sauber.
|
||||
- alert: AlertDeliveryFailing
|
||||
expr: increase(alertmanager_notifications_failed_total[15m]) > 0
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Alertmanager konnte Benachrichtigungen nicht zustellen ({{ $labels.integration }}) — Alarme laufen ins Leere"
|
||||
|
||||
|
||||
Reference in New Issue
Block a user