VersionControl:PRs/maybelle-config/271

maybelle-config#271: maybelle: disk alarm, backup space guard, verified dumps

State	🟣 merged 2026-11-03 12:40:35
Worked on by	magent, justin
Moods	#ops, #tools, #emergence
Deploy	redeploy maybelle (the backup script and the cron-health job).
Branch	maybelle-disk-guard → production, by magent-cryptograss
Opened · last pushed	2026-11-02 22:46:56 · 2026-11-02 22:46:51

On 2 Nov /mnt/persist (40G) reached 100% and memory search went down, with no warning beforehand.

- **cron-health:** a new `maybelle disk` stage fails at 90% on /mnt/persist and lists the biggest directories. Jenkins failures still reach nobody until the alarm route is decided (on the #ops todo), but the stage is red where anyone looking will see it.
- **backup-postgres.sh:** skips the dump when there's less than twice the last dump free (2G with none), and logs it. A dump whose `pg_dump` failed, or that doesn't read back to the end with `pg_restore -f /dev/null`, is deleted, so a truncated one can't become the daily or the offsite copy.
- **Retention:** two-hourly dumps kept 12 hours instead of 24 (`-mmin +720`), about 6G back. Dailies unchanged (7 days).

Checked: YAML parses, the extracted script passes `bash -n`, and the free-space arithmetic gives 2x the last dump in KB. Not run on maybelle.

**Deploy:** redeploy maybelle (the backup script and the cron-health job).
**Worked on by:** magent, Justin, in #ops, #tools and #emergence