Restarting k3s took kube-state-metrics and alloy's log tailers down with it, so
kube_pod_container_info went empty in prometheus. The derivation asked, got a
clean response with zero rows, and counted it as success. Cluster targets went
from 39 to none, the total from 54 to 12, and the scan loop deleted every report
whose target had vanished. Coverage then read 1.0.
None of the four rules fired, and each for a defensible reason: the set was not
empty because the operating host still answered, and every timestamp was fresh
because an empty success updates it. The gap sat exactly between them.
A source that has delivered before and now delivers nothing is treated as a
failure: its previous targets are kept, its timestamp ages, and the stale rule
takes over. A fifth rule watches the total for a drop of more than 40 percent,
and it deliberately also fires on a deliberate shrink — losing 40 percent of the
checked estate is worth a line either way.
The six rancher decisions are gone too. They described versions that no longer
run: the k3s patch took all fifteen of their criticals with it.
Sixty-three criticals on running images have no fix to take. Writing them into
a trivy ignore file would have been the obvious move and the wrong one: trivy
drops ignored findings from its output, so afterwards 'zero because fixed' and
'zero because we looked away' render identically. Every finding stays in
trivy_vuln_info. The decisions sit beside them in entscheidungen.json and are
counted, not subtracted.
Each entry names its CVEs one by one. A blanket entry per image would also
swallow the next finding that shows up there, which is the finding you would
most want to see. The loader rejects an entry without ids, and rejects a review
date it cannot parse — rejecting the whole file, because a half-read decision
list is worse than none.
An unreadable file leaves everything counted as open. Getting that direction
backwards would mean a typo reads as 'all decided', and nobody would notice.
The end-to-end run caught the same mistake in the other half: when the derived
target set is empty the set is unknown, not empty, so the open count now falls
back to every report rather than to zero.
Three python services move off the debian base while we are here — 3.13-slim
carried four criticals with no fix, 3.13-alpine none. All three run on the
stdlib alone and TLS was checked inside the image before the switch.
Gate 3 planned three rules; there are four. The fourth covers a case the others
miss entirely: every source answers cleanly but empty. Then nothing is missing,
because the desired set is empty, the freshness stamps are current, and nothing
is scanned at all. The Python suite already carries that case as "an empty set
is not the same as success", so the rule belongs with it.
These are the first rule unit tests in this stack. Each rule has a case where it
must fire and one where it must stay silent, because a rule that always fires
cannot be told from a correct one otherwise. Two sabotages confirm the tests
bite: an unreachable threshold on the source-freshness rule makes the expected
alert vanish, and removing the six hour grace period makes the missing-targets
rule fire at five hours where the test demands silence.
The grace period is not padding. A full round over roughly 65 images takes time,
so right after a deploy the gap is real rather than wrong.
The dashboard gains coverage and unscanned-image counters in the two free slots
of the top row, and a source-freshness bar at the bottom, so no existing panel
moves. That bar is the only place where a failed derivation can be told apart
from success.
A second monthly probe exists (restore-drill-media). Its failure was
already covered - BackupJobFailed matches the prefix - but its silence was
not, and silence is the failure this rule set was written against.
RestoreDrillStale now matches the prefix instead of one exact name, and
names the affected probe in the message: with two drills, "the restore
probe has not run" no longer says which. BackupCronJobMissing gains the new
cronjob, because a series that disappears is not an alert in Prometheus,
it is quiet.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F2Q4Ri8NGwyTZzScvKnWFM
The exporter pruned first_seen on every scrape, keeping only what it had just
seen. A report that failed to parse - a file being written, a brief I/O error -
was skipped by a silent `continue`, so its findings never entered seen_keys and
their first-seen timestamps were deleted for good. Nothing reported it, and
"first seen" simply restarted at now.
Pruning is now limited to targets whose report was actually read this round.
Proven both ways against a throwaway results directory rather than by reasoning:
make one report unreadable and its entry survives while read_errors counts 1; fix
the other report but drop its finding and that entry is pruned as before. The
distinction is the point - the old behaviour was not too aggressive, it was
indiscriminate.
Two numbers now leave the exporter: trivy_reports_total and
trivy_report_read_errors, with alerts on both. They cover what TrivyScanStale
cannot reach by construction - a target that never produced a report has no series
for time() to compare against, so it stays quiet no matter how long it has been
broken.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Alertmanager was configured as an alerting target but never scraped, so its own
metrics were absent: a silently breaking alert chain could not report itself —
the same blind spot as a missing series, now at the end of the chain. Adds the
operating_alertmanager scrape job and AlertDeliveryFailing on
alertmanager_notifications_failed_total.
The README still claimed alert delivery was deliberately muted via a
room=security null receiver. That route is gone; alertmanager.yml routes
everything to the matrix receiver, so the backup alerts added yesterday do get
delivered. Documentation asserting the opposite is dangerous in both directions,
so it now states the current wiring and keeps the alert-storm history as
background.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verification on CFGMON showed RestoreDrillStale could never fire: restore-drill
has no last_schedule_time series until its first scheduled run (a manually
triggered job does not set it), and an expression over a missing series yields
nothing. BackupNotRunning shares the flaw — deleting a CronJob removes the very
series the alert reads, so it goes quiet instead of firing.
Fall back to kube_cronjob_created, but aggregate with max by(namespace, cronjob):
'or' matches including __name__, so a bare fallback would return BOTH series and
the never-updating created timestamp would fire permanently once the window
elapsed. Add BackupCronJobMissing so a vanished CronJob is itself the alert.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was no rule covering backups at all: a failed nightly job would have gone
unnoticed, which is precisely the silent failure management #0030 is about.
BackupJobFailed catches a failed run, BackupNotRunning catches a CronJob that
stopped scheduling, and RestoreDrillStale fires when the monthly drill stops —
an unverified backup is an assumption again, so the absence of the check is
itself worth alerting on.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Entscheidung sorb 2026-08-01 (Option 1 aus #51): Alarme als
count by (target, severity) statt pro CVE (~58 Serien statt ~1200),
CVE-Details bleiben im Dashboard (trivy_vuln_info unveraendert).
Receiver: inkrementelles save_state nach jedem Alarm, 1s-Sende-Drossel
(Synapse rc_message), recent_resolved-Dedup gegen doppelte Fallback-Haken
bei Batch-Retries, Teilfehler -> 502 liefert nur den Rest nach.
Stumm-Route + Null-Receiver entfernt - Zustellung wieder scharf.
promtool/amtool/py_compile gruen. UNGETESTET bis Deploy (Uebergabe-Issue).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PKhFj1S3UdD6xL2fbWPeYj
- cve-scan: Trivy-Loop ueber die 29 real deployten Images (Cluster-Inventur
2026-08-01 + Prod-Web-Image); 24h-Intervall, Fehler einzelner Images
blockieren nicht
- cve-exporter: Stdlib-Exporter mit first_seen-State (Zeitstrahl), Schema
trivy_vuln_info/_count/_first_seen/_last_scan gemaess Pflichtfeldern
- 3 Alertregeln (CRITICAL sofort, HIGH mit 24h-Daempfung, Scan-Frische) -
promtool SUCCESS 9 rules; alle mit room=security
- matrix-alerts: Label-basiertes Raum-Routing (MATRIX_ROOM_<NAME>), Edits
landen im richtigen Raum via State
- Grafana-Dashboard cve-overview: Severity-Stats, CVE-Tabelle mit
NVD-Link/Fix-Version/first-seen, Zeitstrahl, Verlauf
UNGETESTET bis zum Deploy auf CFGMON (compose up -d).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PKhFj1S3UdD6xL2fbWPeYj
Regeln decken die real erlebten Fehlerklassen ab: TargetDown (coturn-Klasse),
KubePodRestartLoop (36k-Restarts-Klasse), OOM-Kills, RAM-/Swap-/Disk-Druck.
Zustellung in den wartung-Raum ueber einen minimalen Stdlib-Webhook-Receiver
(gleiche Machart wie maintenance-notify, gitops Issue #24); Bot-Token kommt
beim Deploy per .env (Vorlage in .env.example).
UNGETESTET/DEPLOY-PENDING: promtool/amtool-Lint auf dem Mac an haengendem
Docker-Hub-Pull gescheitert - vor dem Deploy auf CFGMON ausfuehren (Images
liegen dort bereits):
docker run --rm -v $PWD/monitoring/prometheus:/cfg:ro --entrypoint promtool prom/prometheus:v3.3.1 check rules /cfg/alerts.yml
docker run --rm -v $PWD/monitoring/alertmanager:/cfg:ro --entrypoint amtool prom/alertmanager:v0.28.1 check-config /cfg/alertmanager.yml
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Thore Cimbal
sorbandClaude Opus 4.8