Restarting k3s took kube-state-metrics and alloy's log tailers down with it, so
kube_pod_container_info went empty in prometheus. The derivation asked, got a
clean response with zero rows, and counted it as success. Cluster targets went
from 39 to none, the total from 54 to 12, and the scan loop deleted every report
whose target had vanished. Coverage then read 1.0.
None of the four rules fired, and each for a defensible reason: the set was not
empty because the operating host still answered, and every timestamp was fresh
because an empty success updates it. The gap sat exactly between them.
A source that has delivered before and now delivers nothing is treated as a
failure: its previous targets are kept, its timestamp ages, and the stale rule
takes over. A fifth rule watches the total for a drop of more than 40 percent,
and it deliberately also fires on a deliberate shrink — losing 40 percent of the
checked estate is worth a line either way.
The six rancher decisions are gone too. They described versions that no longer
run: the k3s patch took all fifteen of their criticals with it.
I wrote 'and ziel in soll' into that condition this evening, and it means a
decision only counts as stale while its image is still deployed. The moment an
image leaves the inventory — a version bump, which is exactly when decisions go
stale — the entry stops being checked and sits in the file forever. Grafana
12.0.0 and 12.4.9 proved it hours later: both entries survived the jump to
13.2.0 and the counter reported zero.
The cost of removing the guard is that a decision made before the scanner reaches
its image shows up here until the next round. The alert waits 24 hours, which
covers it.
The existing arithmetic test could not have caught this, because it reimplements
the calculation and the faulty condition was never in the copy. New tests drive
the real collect() instead, including one that the old behaviour fails.
Both dead grafana entries removed: 28 decisions down to 26.
Sixty-three criticals on running images have no fix to take. Writing them into
a trivy ignore file would have been the obvious move and the wrong one: trivy
drops ignored findings from its output, so afterwards 'zero because fixed' and
'zero because we looked away' render identically. Every finding stays in
trivy_vuln_info. The decisions sit beside them in entscheidungen.json and are
counted, not subtracted.
Each entry names its CVEs one by one. A blanket entry per image would also
swallow the next finding that shows up there, which is the finding you would
most want to see. The loader rejects an entry without ids, and rejects a review
date it cannot parse — rejecting the whole file, because a half-read decision
list is worse than none.
An unreadable file leaves everything counted as open. Getting that direction
backwards would mean a typo reads as 'all decided', and nobody would notice.
The end-to-end run caught the same mistake in the other half: when the derived
target set is empty the set is unknown, not empty, so the open count now falls
back to every report rather than to zero.
Three python services move off the debian base while we are here — 3.13-slim
carried four criticals with no fix, 3.13-alpine none. All three run on the
stdlib alone and TLS was checked inside the image before the switch.
The scan loop is untouched and still reads images.txt, so nothing about this
deployment behaves differently. What changes is that the exporter now knows what
*should* be scanned and can say how much of it is: three sources, none of them
new infrastructure. The cluster and the operating host both already sit in the
same Prometheus, and the registry answers an anonymous token — the same token
dance Trivy performs to pull.
The coverage numbers are the point of this slice. Once deployed they have to
read 24 missing, 2 orphaned and about 0.52, because that is what was counted by
hand on 2026-08-21. A different answer means the derivation is wrong, not the
hand count stale.
Failure handling is the substance rather than an afterthought. A source that
fails costs only its own share; its freshness timestamp keeps ageing instead of
disappearing, because a series that vanishes can never fire a rule — the third
finding of the 2026-08-01 AAR. When every source fails the target file is left
untouched, so a restart during a Prometheus outage cannot clear the estate.
Twenty-one assertions cover it, each paired with its counter-proof: names that
must collapse and names that must not, a source filter that is shown to matter
by removing it, and time-ordered tag selection against the name ordering that
would silently drop v0.10.0. Sabotaging the normalisation turns eleven of them
red, so the suite demonstrably can fail.
The exporter pruned first_seen on every scrape, keeping only what it had just
seen. A report that failed to parse - a file being written, a brief I/O error -
was skipped by a silent `continue`, so its findings never entered seen_keys and
their first-seen timestamps were deleted for good. Nothing reported it, and
"first seen" simply restarted at now.
Pruning is now limited to targets whose report was actually read this round.
Proven both ways against a throwaway results directory rather than by reasoning:
make one report unreadable and its entry survives while read_errors counts 1; fix
the other report but drop its finding and that entry is pruned as before. The
distinction is the point - the old behaviour was not too aggressive, it was
indiscriminate.
Two numbers now leave the exporter: trivy_reports_total and
trivy_report_read_errors, with alerts on both. They cover what TrivyScanStale
cannot reach by construction - a target that never produced a report has no series
for time() to compare against, so it stays quiet no matter how long it has been
broken.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- cve-scan: Trivy-Loop ueber die 29 real deployten Images (Cluster-Inventur
2026-08-01 + Prod-Web-Image); 24h-Intervall, Fehler einzelner Images
blockieren nicht
- cve-exporter: Stdlib-Exporter mit first_seen-State (Zeitstrahl), Schema
trivy_vuln_info/_count/_first_seen/_last_scan gemaess Pflichtfeldern
- 3 Alertregeln (CRITICAL sofort, HIGH mit 24h-Daempfung, Scan-Frische) -
promtool SUCCESS 9 rules; alle mit room=security
- matrix-alerts: Label-basiertes Raum-Routing (MATRIX_ROOM_<NAME>), Edits
landen im richtigen Raum via State
- Grafana-Dashboard cve-overview: Severity-Stats, CVE-Tabelle mit
NVD-Link/Fix-Version/first-seen, Zeitstrahl, Verlauf
UNGETESTET bis zum Deploy auf CFGMON (compose up -d).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PKhFj1S3UdD6xL2fbWPeYj
Thore Cimbal
