VersionControl:PRs/maybelle-config/271: Difference between revisions
From PickiPedia: A knowledge base of bluegrass, old time psychedelic jams, and other public domain music
Jump to navigationJump to search
cryptograss/maybelle-config#271, as GitHub has it |
cryptograss/maybelle-config#271, as GitHub has it |
||
| (One intermediate revision by the same user not shown) | |||
| Line 5: | Line 5: | ||
|url=https://github.com/cryptograss/maybelle-config/pull/271 | |url=https://github.com/cryptograss/maybelle-config/pull/271 | ||
|author=magent-cryptograss | |author=magent-cryptograss | ||
|state= | |state=merged | ||
|base=production | |base=production | ||
|branch=maybelle-disk-guard | |branch=maybelle-disk-guard | ||
|opened=2026-11-02 22:46:56 | |opened=2026-11-02 22:46:56 | ||
|pushed=2026-11-02 22:46:51 | |pushed=2026-11-02 22:46:51 | ||
|merged= | |merged=2026-11-03 12:40:35 | ||
|closed= | |closed= | ||
|worked_on_by=magent, justin | |worked_on_by=magent, justin | ||
| Line 16: | Line 16: | ||
|deploy=redeploy maybelle (the backup script and the cron-health job). | |deploy=redeploy maybelle (the backup script and the cron-health job). | ||
}} | }} | ||
<pre class="pr-body">On 2 Nov /mnt/persist (40G) reached 100% and memory search went down, with no warning beforehand. | |||
- **cron-health:** a new `maybelle disk` stage fails at 90% on /mnt/persist and lists the biggest directories. Jenkins failures still reach nobody until the alarm route is decided (on the #ops todo), but the stage is red where anyone looking will see it. | |||
- **backup-postgres.sh:** skips the dump when there's less than twice the last dump free (2G with none), and logs it. A dump whose `pg_dump` failed, or that doesn't read back to the end with `pg_restore -f /dev/null`, is deleted, so a truncated one can't become the daily or the offsite copy. | |||
- **Retention:** two-hourly dumps kept 12 hours instead of 24 (`-mmin +720`), about 6G back. Dailies unchanged (7 days). | |||
Checked: YAML parses, the extracted script passes `bash -n`, and the free-space arithmetic gives 2x the last dump in KB. Not run on maybelle. | |||
**Deploy:** redeploy maybelle (the backup script and the cron-health job). | |||
**Worked on by:** magent, Justin, in #ops, #tools and #emergence</pre> | |||
Latest revision as of 17:06, 9 November 2026
| State | 🟣 merged 2026-11-03 12:40:35 |
|---|---|
| Worked on by | magent, justin |
| Moods | #ops, #tools, #emergence |
| Deploy | redeploy maybelle (the backup script and the cron-health job). |
| Branch | maybelle-disk-guard → production, by magent-cryptograss
|
| Opened · last pushed | 2026-11-02 22:46:56 · 2026-11-02 22:46:51 |
On 2 Nov /mnt/persist (40G) reached 100% and memory search went down, with no warning beforehand. - **cron-health:** a new `maybelle disk` stage fails at 90% on /mnt/persist and lists the biggest directories. Jenkins failures still reach nobody until the alarm route is decided (on the #ops todo), but the stage is red where anyone looking will see it. - **backup-postgres.sh:** skips the dump when there's less than twice the last dump free (2G with none), and logs it. A dump whose `pg_dump` failed, or that doesn't read back to the end with `pg_restore -f /dev/null`, is deleted, so a truncated one can't become the daily or the offsite copy. - **Retention:** two-hourly dumps kept 12 hours instead of 24 (`-mmin +720`), about 6G back. Dailies unchanged (7 days). Checked: YAML parses, the extracted script passes `bash -n`, and the free-space arithmetic gives 2x the last dump in KB. Not run on maybelle. **Deploy:** redeploy maybelle (the backup script and the cron-health job). **Worked on by:** magent, Justin, in #ops, #tools and #emergence