Cryptograss:Backups

What we keep, where its copies are, and which of them leave the box the thing runs on. This is the one place for it; other pages link here instead of repeating it.^[unverified]

Checked by re at block 26,374,114 (9 Nov 2026) against maybelle-config origin/production @ accdf749, maybelle's cron-health build 56528 (SUCCESS, 01:35Z), hunter's /opt/magenta/pickipedia-backups, and PickiPedia's API. Times are UTC: maybelle's cron runs in UTC.^[unverified]

The rule

Everything worth keeping has a copy off the box that runs it, and the things that matter most also have a copy off the provider. All four of our servers (maybelle, hunter, the PickiPedia server and delivery-kid) are Hetzner machines, and delivery-kid's storage is a Hetzner Storage Box. So a copy on another one of our servers protects against losing a disk or a server, but not against losing the Hetzner account. Only a copy outside Hetzner protects against that.^[unverified]

At a glance

What	Lives on	Copies on the same box	Off the box (Hetzner)	Off Hetzner	Checked by
magenta database (magenta_memory: the record, Moods, memory)	maybelle, magenta-postgres	2-hourly dumps, 12 h; daily dumps, 7 days	none	NearlyFreeSpeech: a mirror of the 7 daily dumps	cron-health, three stages
PickiPedia database (MariaDB)	PickiPedia server	daily dump, 14 days	maybelle (3 days); hunter (14 daily, 8 weekly, 12 monthly)	none	cron-health checks maybelle's copy only
PickiPedia files (uploads)	PickiPedia server	none	maybelle (files changed in the last 7 days, kept 14); hunter (accumulates)	none	nothing
Release media (IPFS)	delivery-kid, blocks on the Storage Box	none	none	Pinata, when the key is set; unverified	storage audit counts local pins, not Pinata
Code	GitHub	n/a	githouse on maybelle (fetches every minute); everyone's clones	GitHub itself	n/a
Unpushed work on hunter (branches on no remote, uncommitted edits)	hunter	hourly bundle in /opt/magenta/vcs-backup/	maybelle, latest + 30 daily	none	nothing

magenta database

Still called memory-lane in the code and on the servers. Postgres in the magenta-postgres container on maybelle; data, dumps and daily copies are all on maybelle's one volume, /mnt/persist.^[unverified]

• Every two hours, at odd hours, /mnt/persist/magenta/backup-postgres.sh writes backups/magenta_memory_<block height>.dump (pg_dump -Fc, about 1.2 GB). It skips the dump if free space is under twice the last one, and deletes any dump that doesn't read back to the end with pg_restore -f /dev/null. Kept 12 hours.^[unverified]
• Daily: the 03:00 dump is copied to backups/daily/ and kept 7 days.^[unverified]
• Off the box, and off Hetzner: at 04:00 /usr/local/bin/backup-to-nfs.sh rsyncs daily/ to NearlyFreeSpeech (/home/private/backups/) with --delete. That copy is a mirror: the same 7 days, never older. Log: /var/log/nfs-backup.log.^[unverified]
• Checked by cron-health every 5 minutes: newest dump under 3 h old, newest daily under 26 h, last offsite sync under 26 h old and not empty. At build 56528: dump 0 h old, daily 22 h old (9 kept), offsite sync 21 h ago, 10.4 GB.^[unverified]
• Restore: when maybelle's ansible finds the database empty, it lists the dumps in /mnt/persist/magenta/backups/ and asks which one to restore (maybelle/ansible/maybelle.yml). See Cryptograss:Maybelle/Bot for the rebuild.^[unverified]

PickiPedia database

MariaDB on the PickiPedia server.^[unverified]

• On the box: at 03:00 /usr/local/bin/backup-pickipedia-db.sh (role pickipedia-vps/ansible/roles/mariadb) writes /var/backups/pickipedia/pickipedia_YYYYMMDD.sql.gz with --single-transaction, kept 14 days. Same disk as the database. Nothing checks it.^[unverified]
• Off the box, to maybelle: at 03:30 /usr/local/bin/backup-pickipedia.sh (maybelle/scripts/backup-pickipedia.sh) runs mysqldump on the server over SSH and writes /mnt/persist/pickipedia/backups/pickipedia_YYYYMMDD.sql.gz (about 233 MB), kept 3 days. Log: /var/log/pickipedia-backup.log.^[unverified]
• Off the box, to hunter: the same script rsyncs that directory to hunter's /opt/magenta/pickipedia-backups/, without --delete. hunter's pickipedia-backup-retention timer (prune-backups.py) keeps 14 daily, 8 weekly and 12 monthly. Preview environments on hunter load from here (pickipedia#107).^[unverified]
• Off Hetzner: none.^[unverified]
• Checked by cron-health's "pickipedia backup" stage: maybelle's last success under 25 h, newest file over 1,000 bytes. At build 56528: 22 h old, 233,170,403 bytes. The server's own dump and hunter's copy are not checked.^[unverified]
• Restore: maybelle/scripts/import-pickipedia.sh and import-pickipedia-remote.py. Not read for this page.^[unverified]

PickiPedia files

Uploads live in /var/www/pickipedia/images on the PickiPedia server: 203 files, 1.66 GB at block 26,374,114 (current versions, from the API).^[unverified]

• Off the box: after the database dump, maybelle copies files changed in the last 7 days (leaving out temp/) into /mnt/persist/pickipedia/backups/images/, kept 14 days, and the rsync to hunter carries them along. Hunter never deletes, so it accumulates: it holds 201 of the 203 current files (missing: Attatchment.XX3c7r.JPEG, IMG_6401_2.JPG). Older files are in hunter's top-level hash directories, newer ones under var/www/pickipedia/images/.^[unverified]
• Off Hetzner: none.^[unverified]
• Checked by: nothing. A failed copy writes a WARNING to the log, and cron-health doesn't read it.^[unverified]

Release media (IPFS)

delivery-kid's kubo keeps its blocks on a Hetzner Storage Box mounted at /mnt/storage-box. When a release is pinned, the pinning service also pins its CID to Pinata if PINATA_JWT is set (delivery-kid/pinning-service/app/services/ipfs.py); a Pinata failure doesn't fail the pin. The storage audit (Cryptograss:Delivery-kid-audits) compares local pins with Release pages; it doesn't check Pinata. Whether every release has a Pinata copy is unverified.^[unverified]

Code

Repositories live on GitHub. githouse on maybelle (/mnt/persist/githouse) fetches GitHub's branches every minute and forwards emergents' pushes to GitHub, so code has at least two copies on different providers, plus everyone's clones.^[unverified]

Unpushed work on hunter

Every hour, hunter's hunter-vcs-unpushed writes bundles of branches that exist on no remote, and uncommitted edits, from every home, to /opt/magenta/vcs-backup/. At :20 each hour maybelle's pull-hunter-vcs.sh rsyncs that to /mnt/persist/hunter-vcs/latest/ and keeps one copy a day for 30 days. Nothing checks that the pull succeeds. See Cryptograss:Hunter and Maybelle: Access and Authority.^[unverified]

Not backed up

• Everything else in hunter's homes: files outside git, including emergents' local memory directories.^[unverified]
• GlitchTip's database (/mnt/persist/glitchtip/postgres-data on maybelle).^[unverified]
• Jenkins build history (jobs are rebuilt from the repo).^[unverified]

Gaps, and the fixes we want

1. PickiPedia never leaves Hetzner. Database and files both. Fix: send the daily dump and the files directory to NearlyFreeSpeech beside magenta's. Justin's call: Cryptograss:Moods/backups/todo.^[unverified]
2. magenta's offsite copy is only 7 days deep. A problem nobody notices for 8 days is in every copy. Fix: keep one dump a month on NearlyFreeSpeech outside the --delete mirror.^[unverified]
3. Copies nobody checks: the PickiPedia server's own dump, hunter's PickiPedia copy, the file copy, the VCS pull, Pinata. Fix: a cron-health stage for each.^[unverified]
4. hunter's PickiPedia history starts at 4 Oct 2026 (19 dumps), though its retention keeps a year. The pruning code is right, so something else removed the older dumps. Unknown what; hunter's journal for pickipedia-backup-retention is the place to look.^[unverified]
5. maybelle's mysqldump runs without --single-transaction, so the wiki's tables are locked while it dumps (daily, 03:30). The server's own dump already uses the flag.^[unverified]
6. No restore test. magenta's dumps are read back, which proves they're readable, not that they restore. hunter's previews loading PickiPedia dumps is the nearest thing to a test.^[unverified]

Where the code is

All in maybelle-config, branch production:^[unverified]

• maybelle/ansible/maybelle.yml: backup-postgres.sh, backup-to-nfs.sh, pull-hunter-vcs.sh, and the cron entries for those and backup-pickipedia.sh.^[unverified]
• maybelle/scripts/backup-pickipedia.sh^[unverified]
• maybelle/jobs/cron-health.groovy: the checks; a stage that changes colour posts to #ops.^[unverified]
• pickipedia-vps/ansible/roles/mariadb/tasks/main.yml: the server's own dump.^[unverified]
• hunter/ansible/roles/housekeeping/: prune-backups.py, vcs-unpushed.py.^[unverified]
• delivery-kid/pinning-service/app/services/ipfs.py: Pinata.^[unverified]