The exporter pruned first_seen on every scrape, keeping only what it had just
seen. A report that failed to parse - a file being written, a brief I/O error -
was skipped by a silent `continue`, so its findings never entered seen_keys and
their first-seen timestamps were deleted for good. Nothing reported it, and
"first seen" simply restarted at now.
Pruning is now limited to targets whose report was actually read this round.
Proven both ways against a throwaway results directory rather than by reasoning:
make one report unreadable and its entry survives while read_errors counts 1; fix
the other report but drop its finding and that entry is pruned as before. The
distinction is the point - the old behaviour was not too aggressive, it was
indiscriminate.
Two numbers now leave the exporter: trivy_reports_total and
trivy_report_read_errors, with alerts on both. They cover what TrivyScanStale
cannot reach by construction - a target that never produced a report has no series
for time() to compare against, so it stays quiet no matter how long it has been
broken.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Alertmanager was configured as an alerting target but never scraped, so its own
metrics were absent: a silently breaking alert chain could not report itself —
the same blind spot as a missing series, now at the end of the chain. Adds the
operating_alertmanager scrape job and AlertDeliveryFailing on
alertmanager_notifications_failed_total.
The README still claimed alert delivery was deliberately muted via a
room=security null receiver. That route is gone; alertmanager.yml routes
everything to the matrix receiver, so the backup alerts added yesterday do get
delivered. Documentation asserting the opposite is dangerous in both directions,
so it now states the current wiring and keeps the alert-storm history as
background.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verification on CFGMON showed RestoreDrillStale could never fire: restore-drill
has no last_schedule_time series until its first scheduled run (a manually
triggered job does not set it), and an expression over a missing series yields
nothing. BackupNotRunning shares the flaw — deleting a CronJob removes the very
series the alert reads, so it goes quiet instead of firing.
Fall back to kube_cronjob_created, but aggregate with max by(namespace, cronjob):
'or' matches including __name__, so a bare fallback would return BOTH series and
the never-updating created timestamp would fire permanently once the window
elapsed. Add BackupCronJobMissing so a vanished CronJob is itself the alert.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was no rule covering backups at all: a failed nightly job would have gone
unnoticed, which is precisely the silent failure management #0030 is about.
BackupJobFailed catches a failed run, BackupNotRunning catches a CronJob that
stopped scheduling, and RestoreDrillStale fires when the monthly drill stops —
an unverified backup is an assumption again, so the absence of the check is
itself worth alerting on.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Entscheidung sorb 2026-08-01 (Option 1 aus #51): Alarme als
count by (target, severity) statt pro CVE (~58 Serien statt ~1200),
CVE-Details bleiben im Dashboard (trivy_vuln_info unveraendert).
Receiver: inkrementelles save_state nach jedem Alarm, 1s-Sende-Drossel
(Synapse rc_message), recent_resolved-Dedup gegen doppelte Fallback-Haken
bei Batch-Retries, Teilfehler -> 502 liefert nur den Rest nach.
Stumm-Route + Null-Receiver entfernt - Zustellung wieder scharf.
promtool/amtool/py_compile gruen. UNGETESTET bis Deploy (Uebergabe-Issue).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PKhFj1S3UdD6xL2fbWPeYj
- cve-scan: Trivy-Loop ueber die 29 real deployten Images (Cluster-Inventur
2026-08-01 + Prod-Web-Image); 24h-Intervall, Fehler einzelner Images
blockieren nicht
- cve-exporter: Stdlib-Exporter mit first_seen-State (Zeitstrahl), Schema
trivy_vuln_info/_count/_first_seen/_last_scan gemaess Pflichtfeldern
- 3 Alertregeln (CRITICAL sofort, HIGH mit 24h-Daempfung, Scan-Frische) -
promtool SUCCESS 9 rules; alle mit room=security
- matrix-alerts: Label-basiertes Raum-Routing (MATRIX_ROOM_<NAME>), Edits
landen im richtigen Raum via State
- Grafana-Dashboard cve-overview: Severity-Stats, CVE-Tabelle mit
NVD-Link/Fix-Version/first-seen, Zeitstrahl, Verlauf
UNGETESTET bis zum Deploy auf CFGMON (compose up -d).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PKhFj1S3UdD6xL2fbWPeYj
Regeln decken die real erlebten Fehlerklassen ab: TargetDown (coturn-Klasse),
KubePodRestartLoop (36k-Restarts-Klasse), OOM-Kills, RAM-/Swap-/Disk-Druck.
Zustellung in den wartung-Raum ueber einen minimalen Stdlib-Webhook-Receiver
(gleiche Machart wie maintenance-notify, gitops Issue #24); Bot-Token kommt
beim Deploy per .env (Vorlage in .env.example).
UNGETESTET/DEPLOY-PENDING: promtool/amtool-Lint auf dem Mac an haengendem
Docker-Hub-Pull gescheitert - vor dem Deploy auf CFGMON ausfuehren (Images
liegen dort bereits):
docker run --rm -v $PWD/monitoring/prometheus:/cfg:ro --entrypoint promtool prom/prometheus:v3.3.1 check rules /cfg/alerts.yml
docker run --rm -v $PWD/monitoring/alertmanager:/cfg:ro --entrypoint amtool prom/alertmanager:v0.28.1 check-config /cfg/alertmanager.yml
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Grafana-Provisioning ergaenzt: Datasources (prometheus/loki) mit den
bestehenden UIDs, damit alle Dashboards weiter funktionieren, sowie vier
Dashboard-Provider passend zu den vorhandenen Ordnern.
- 9 Dashboards aus der laufenden Instanz exportiert und bereinigt: id/version
entfernt, hart kodierte und verwaiste Datasource-UIDs auf die
provisionierten UIDs umgeschrieben.
- Images von :latest auf feste Versionen gepinnt (Prometheus v3.3.1,
Loki 3.7.1, Alloy v1.16.0, Grafana 12.0.0, node-exporter v1.9.1).
- Alloy: Docker-Metadaten (container, compose_service, compose_project) als
Loki-Labels via discovery.relabel.
- .env-Variablen auf GRAFANA_ADMIN_USER/GRAFANA_ADMIN_PASSWORD vereinheitlicht,
.env.example ergaenzt, .env bleibt ungetrackt.
- prometheus.yml kommentiert (Remote-Write-Quellen, Fremd-Stack-Targets),
toten auskommentierten Matrix-Job entfernt.
- README mit Deployment, Struktur, externen Abhaengigkeiten und offenen
Sicherheitspunkten (offene Ports 9090/3100/9100).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Import des bisherigen Stands aus /opt/monitoring auf dem Operating-Host,
unveraendert. Images auf :latest, Grafana ohne Provisioning (Datasources und
Dashboards nur in der grafana.db), keine Dokumentation.
Dient als Ausgangspunkt fuer den Umbau auf einen vollstaendig
provisionierbaren Stack.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Thore CimbalandClaude Opus 5
sorbandClaude Opus 4.8
sorbandClaude Opus 5