Deploying e0808ba surfaced the trap in practice: docker pins a single-file bind
mount to the inode, git pull replaces files by rename, so the container kept
serving the old alerts.yml while SIGHUP reported a successful reload — host and
container md5 differed and BackupCronJobMissing simply was not there.
The trap was already documented in this README, in detail, with the correct
command and a verification snippet, and it still bit. A footgun you avoid only by
reading gets stepped on eventually, so remove it structurally: directory mounts
resolve through the path on every access. Config paths are unchanged, so
--config.file keeps working. Remaining single-file mounts (loki, alloy, the
scripts) are named in the README as still needing --force-recreate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verification on CFGMON showed RestoreDrillStale could never fire: restore-drill
has no last_schedule_time series until its first scheduled run (a manually
triggered job does not set it), and an expression over a missing series yields
nothing. BackupNotRunning shares the flaw — deleting a CronJob removes the very
series the alert reads, so it goes quiet instead of firing.
Fall back to kube_cronjob_created, but aggregate with max by(namespace, cronjob):
'or' matches including __name__, so a bare fallback would return BOTH series and
the never-updating created timestamp would fire permanently once the window
elapsed. Add BackupCronJobMissing so a vanished CronJob is itself the alert.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was no rule covering backups at all: a failed nightly job would have gone
unnoticed, which is precisely the silent failure management #0030 is about.
BackupJobFailed catches a failed run, BackupNotRunning catches a CronJob that
stopped scheduling, and RestoreDrillStale fires when the monthly drill stops —
an unverified backup is an assumption again, so the absence of the check is
itself worth alerting on.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sorbandClaude Opus 4.8