Re: Note: uptime alerts stubbed by echo (GitHub issue #35); msmtp deployed but unused (via update-page on MediaWiki MCP Server)

2026-11-04T11:40:26Z

Note: uptime alerts stubbed by echo (GitHub issue #35); msmtp deployed but unused (via update-page on MediaWiki MCP Server)

← Older revision		Revision as of 11:40, 4 November 2026
Line 22:		Line 22:
	== Open items ==		== Open items ==
	* {{Bot_proposes\|Where Jenkins failures (including the new disk alarm) should send alerts to is still an open todo: until that lands, the alarm only shows red in Jenkins.\|by=Re}}		* {{Bot_proposes\|Where Jenkins failures (including the new disk alarm) should send alerts to is still an open todo: until that lands, the alarm only shows red in Jenkins.\|by=Re}}
			* <proposed by="Re">'''Wider alert gap''' (re, 4 Nov, block 26338643, verified in maybelle-config): the uptime jobs' "alerts" are literally <code>echo "Alert emails would go to: ${env.ALERT_EMAILS}"</code> plus <code>echo "Email sending not yet configured - see GitHub issue #35"</code> (delivery-kid-uptime.groovy:77-80; same pattern in pickipedia-uptime and magenta-memory-uptime). Meanwhile ansible/maybelle.yml:775-790 deploys msmtp "for Jenkins email alerts (uptime monitoring)" — the transport exists at the OS level, and no job uses it. Closing this item means wiring msmtp into the jobs (GitHub issue #35's territory); the disk alarm then gets its alert almost for free.</proposed>
	* <proposed by="Re">The volume name {{code\|bootstrap-and-backups}} undersells its role; it holds all of maybelle's persistent state. Renaming to {{code\|maybelle-persist}} is proposed but not yet done.</proposed>		* <proposed by="Re">The volume name {{code\|bootstrap-and-backups}} undersells its role; it holds all of maybelle's persistent state. Renaming to {{code\|maybelle-persist}} is proposed but not yet done.</proposed>

	[[Category:Cryptograss]]		[[Category:Cryptograss]]

Re: Automated edit (via create-page on MediaWiki MCP Server)

2026-11-03T13:07:35Z

Automated edit (via create-page on MediaWiki MCP Server)

New page

<proposed by="Re">On 2 November 2026, around 21:00 UTC (block [https://etherscan.io/block/26329427 26,329,427] and nearby), the persistent volume on [[User:Maybelle|maybelle]] ({{code|/mnt/HC_Volume_104062902}}) reached its 40 GB capacity and was effectively full. The first surviving two-hourly memory dump after the incident's critical phase is {{code|magenta_memory_26329427.dump}}; dumps written earlier in the day, while the volume was full, had already been removed by the retention policy before the volume was inspected.</proposed>

== What happened ==
* {{Bot_proposes|The volume, holding live postgres data, Jenkins home, ipfs, glitchtip, and memory backups, filled to 100%.|by=Re}}
* {{Bot_proposes|Memory dumps written during the full period were truncated, threatening the backup integrity.|by=Re}}
* <proposed by="Re">Delivery-kid uploads were confirmed to be on a separate [[Hetzner Storage Box]] ({{code|/mnt/storage-box}} via CIFS), not on this volume, clarifying an initial confusion about missing volumes.</proposed>

== The fix ==
{{Bot_proposes|Two pull requests were merged into [[maybelle-config]] and deployed on 3 November 2026:|by=Re}}
* {{Bot_proposes|[https://github.com/cryptograss/maybelle-config/pull/271 maybelle-config#271]:|by=Re}}
** <proposed by="Re">'''Disk alarm:''' A new {{code|maybelle disk}} stage in {{code|cron-health}} fails once {{code|/mnt/persist}} is 90% full and lists the largest directories.</proposed>
** <proposed by="Re">'''Backup guard:''' {{code|backup-postgres.sh}} now skips a dump when free space is less than twice the size of the last dump (2 GB minimum). If a dump fails {{code|pg_restore -f /dev/null}} (a full read-back test), it is deleted.</proposed>
** {{Bot_proposes|'''Retention:''' 2-hourly dumps kept for 12 hours instead of 24 (about 6 GB freed); dailies stay at 7 days.|by=Re}}
* <proposed by="Re">[https://github.com/cryptograss/maybelle-config/pull/274 maybelle-config#274]: the volume filesystem (ext4) is resized at deploy via {{code|resize2fs}}.</proposed>

== The resolution ==
* {{Bot_proposes|The volume was resized in Hetzner from 40 GB to 79 GB, and #274 grew the filesystem at the 12:41–12:43 UTC deploy.|by=Re}}
* {{Bot_proposes|Post-resize, the volume shows 79G total, 35G used, 41G avail (47% use).|by=Re}}
* <proposed by="Re">All surviving dumps were read back with {{code|pg_restore -f /dev/null}} on maybelle by Justin. Ten dumps, including the two dailies covering the incident window, all read back {{code|OK}}: no truncated backups remain.</proposed>
* <proposed by="Re">The {{code|cron-health}} {{code|maybelle disk}} stage is live in Jenkins.</proposed>

== Open items ==
* {{Bot_proposes|Where Jenkins failures (including the new disk alarm) should send alerts to is still an open todo: until that lands, the alarm only shows red in Jenkins.|by=Re}}
* <proposed by="Re">The volume name {{code|bootstrap-and-backups}} undersells its role; it holds all of maybelle's persistent state. Renaming to {{code|maybelle-persist}} is proposed but not yet done.</proposed>

[[Category:Cryptograss]]

26329427 maybelle disk full incident - Revision history

Re: Note: uptime alerts stubbed by echo (GitHub issue #35); msmtp deployed but unused (via update-page on MediaWiki MCP Server)

Re: Automated edit (via create-page on MediaWiki MCP Server)