Cryptograss:Backups: Difference between revisions
From PickiPedia: A knowledge base of bluegrass, old time psychedelic jams, and other public domain music
Jump to navigationJump to search
Fix redlink: the audits are subpages (via update-page on MediaWiki MCP Server) |
Vault password: the sweep (via update-page on MediaWiki MCP Server) |
||
| (4 intermediate revisions by the same user not shown) | |||
| Line 22: | Line 22: | ||
| '''Unpushed work on hunter''' (branches on no remote, uncommitted edits) || hunter || hourly bundle in <code>/opt/magenta/vcs-backup/</code> || maybelle, latest + 30 daily || '''none''' || '''nothing''' | | '''Unpushed work on hunter''' (branches on no remote, uncommitted edits) || hunter || hourly bundle in <code>/opt/magenta/vcs-backup/</code> || maybelle, latest + 30 daily || '''none''' || '''nothing''' | ||
|} | |} | ||
== Cold start: everything it needs == | |||
{{Bot_proposes|What [[Cryptograss:Drill:Cold Start Rebuild (by block 26,400,000){{!}}the cold start drill]] needs if every Hetzner machine is gone at once, and where each thing comes back from. This is the drill's step one, the property map. Checked by re at block 26,401,540 (12 Nov 2026) against maybelle-config <code>production</code> @ <code>38730290</code>. A row without a copy outside Hetzner is a gap.|by=Re}} | |||
{| class="wikitable" | |||
! Needed !! Lives on !! Copy outside Hetzner !! Status | |||
|- | |||
| '''The vault password''' || people only. maybelle has it only while a deploy runs (see below) || people || who holds it besides Justin is not written anywhere | |||
|- | |||
| maybelle-config, with the encrypted vault (every secret not made on the box) || githouse on maybelle || GitHub (githouse forwards every push and merge) || ✓ | |||
|- | |||
| Every other repository (magenta, pickipedia, arthel, rabbithole, pickipedia-mcp …) || githouse || GitHub || ✓ | |||
|- | |||
| githouse's signed merge log (<code>/srv/git/.merges</code>) || maybelle || none || ✗ small; the merges themselves are on GitHub | |||
|- | |||
| magenta database || maybelle || NearlyFreeSpeech, 7 days || ✓, but only 7 days deep | |||
|- | |||
| PickiPedia database || PickiPedia server || none || ✗ | |||
|- | |||
| PickiPedia files (uploads) || PickiPedia server || none || ✗ | |||
|- | |||
| PickiPedia's code and settings || pickipedia repo; LocalSettings from the vault || GitHub || ✓ | |||
|- | |||
| Release media (IPFS blocks) || delivery-kid's Storage Box || Pinata, if every pin reached it || unverified | |||
|- | |||
| delivery-kid's staging (unpublished uploads, 171 MB at 8 Nov's audit) and seeding directories || Storage Box || none || ✗ unpublished uploads; seeding can be rebuilt from IPFS | |||
|- | |||
| IPFS node identity (<code>/var/lib/ipfs</code>) || delivery-kid || none || a new node gets a new peer ID; CIDs don't change | |||
|- | |||
| Unpushed work on hunter || hunter, pulled to maybelle || none || ✗ the drill counts homes as disposable | |||
|- | |||
| GlitchTip's database || maybelle || none || ✗ error history only | |||
|- | |||
| justinholmes.com (arthel) || NearlyFreeSpeech || it's built from GitHub || ✓ | |||
|- | |||
| DNS || cryptograss.live and justinholmes.com on NearlyFreeSpeech; pickipedia.xyz on Namecheap || the providers themselves || records are not in code: new servers mean new addresses, typed in by hand | |||
|- | |||
| Keys made on the box (githouse's mirror key, the containers' githouse keys, maybelle's backup key) || made by the playbooks || none needed || ✓ made again on deploy; the public halves flow to githouse by themselves | |||
|} | |||
=== The vault password === | |||
{{Bot_proposes|It is not kept on maybelle. <code>maybelle-chapter-1.sh</code> copies it from the deploying person's laptop to <code>/mnt/persist/.vault-password</code>, and the playbook's task "Remove vault password file" deletes it near the end. The six newest maybelle deploy logs (11–12 Nov) list that task as changed, so it was there during the deploy and removed after. The gap: a deploy that failed before that task left the password on the volume until the next one got that far. <code>maybelle-config:re/vault-password-always-removed</code> removes it when the deploy script exits, however the deploy ends. On the same branch, a cron on maybelle (<code>vault-password-sweep.sh</code>, every 10 minutes) removes it if it's there outside a deploy, and cron-health's "vault password sweep" stage then stays red for a day, which posts to #ops. The other targets (hunter, PickiPedia, delivery-kid) write it to <code>/tmp/vault_pass_$$</code> and remove it on exit already. No backup copies the whole volume, so it is in no backup.|by=Re}} | |||
== magenta database == | == magenta database == | ||
| Line 64: | Line 106: | ||
== Gaps, and the fixes we want == | == Gaps, and the fixes we want == | ||
# {{Bot_proposes|'''PickiPedia never leaves Hetzner.''' Database and files both | # {{Bot_proposes|'''PickiPedia never leaves Hetzner.''' Database and files both, and the other ✗ rows under [[#Cold start: everything it needs{{!}}Cold start]]. Fix: one offsite job, to a store outside Hetzner, for everything the cold start needs.|by=Re}} | ||
# {{Bot_proposes|'''magenta's offsite copy is only 7 days deep.''' A problem nobody notices for 8 days is in every copy. Fix: | # {{Bot_proposes|'''magenta's offsite copy is only 7 days deep.''' A problem nobody notices for 8 days is in every copy. Fix: the offsite job keeps a history (7 daily, 4 weekly, 12 monthly), and maybelle's key to the store can add but not delete, so whoever controls maybelle can't empty it either.|by=Re}} | ||
# {{Bot_proposes|'''Copies nobody checks:''' the PickiPedia server's own dump, hunter's PickiPedia copy, the file copy, the VCS pull, Pinata. Fix: a cron-health stage for each.|by=Re}} | # {{Bot_proposes|'''Copies nobody checks:''' the PickiPedia server's own dump, hunter's PickiPedia copy, the file copy, the VCS pull, Pinata. Fix: a cron-health stage for each.|by=Re}} | ||
# {{Bot_proposes|'''hunter's PickiPedia history starts at 4 Oct 2026''' (19 dumps), though its retention keeps a year. The pruning code is right, so something else removed the older dumps. Unknown what; hunter's journal for <code>pickipedia-backup-retention</code> is the place to look.|by=Re}} | # {{Bot_proposes|'''hunter's PickiPedia history starts at 4 Oct 2026''' (19 dumps), though its retention keeps a year. The pruning code is right, so something else removed the older dumps. Unknown what; hunter's journal for <code>pickipedia-backup-retention</code> is the place to look.|by=Re}} | ||
# {{Bot_proposes|maybelle's <code>mysqldump</code> runs without <code>--single-transaction</code>, so the wiki's tables are locked while it dumps (daily, 03:30). The server's own dump already uses the flag.|by=Re}} | # {{Bot_proposes|maybelle's <code>mysqldump</code> runs without <code>--single-transaction</code>, so the wiki's tables are locked while it dumps (daily, 03:30). The server's own dump already uses the flag.|by=Re}} | ||
# {{Bot_proposes|'''No restore test.''' magenta's dumps are read back, which proves they're readable, not that they restore. hunter's previews loading PickiPedia dumps is the nearest thing to a test.|by=Re}} | # {{Bot_proposes|'''No restore test.''' magenta's dumps are read back, which proves they're readable, not that they restore. hunter's previews loading PickiPedia dumps is the nearest thing to a test.|by=Re}} | ||
=== Where the offsite copy goes === | |||
{{Bot_proposes|Prices from each provider's own pricing page, 12 Nov 2026:|by=Re}} | |||
{| class="wikitable" | |||
! Store !! Price !! Outside Hetzner? !! Notes | |||
|- | |||
| NearlyFreeSpeech (today's) || $1.00 per GiB-month (site storage) || yes || magenta's 10.4 GB costs about $10 a month; 30 GB would be about $30 | |||
|- | |||
| Backblaze B2 || $6.95 per TB-month (about $0.007 per GB) || yes || pay for what's stored; object lock and keys that can't delete | |||
|- | |||
| rsync.net || 1.5¢ per GB-month, 800 GB minimum (about $12 a month) || yes || plain SSH and rsync; snapshots | |||
|- | |||
| Hetzner Storage Box BX11 || €3.20 a month for 1 TB || '''no''' || cheapest per TB, but on the same provider | |||
|} | |||
{{Bot_proposes|re's choice: '''Backblaze B2''', with restic: encrypted before it leaves, deduplicated, retention by rule. 30 GB is about $0.21 a month. The restic password and the B2 key go in the vault, so the chain is: a person holds the vault password, GitHub holds the vault, the vault opens B2. Justin's call: [[Cryptograss:Moods/backups/todo]].|by=Re}} | |||
== Where the code is == | == Where the code is == | ||
Latest revision as of 21:36, 12 November 2026
What we keep, where its copies are, and which of them leave the box the thing runs on. This is the one place for it; other pages link here instead of repeating it.[unverified]
Checked by re at block 26,374,114 (9 Nov 2026) against maybelle-config origin/production @ accdf749, maybelle's cron-health build 56528 (SUCCESS, 01:35Z), hunter's /opt/magenta/pickipedia-backups, and PickiPedia's API. Times are UTC: maybelle's cron runs in UTC.[unverified]
The rule
Everything worth keeping has a copy off the box that runs it, and the things that matter most also have a copy off the provider. All four of our servers (maybelle, hunter, the PickiPedia server and delivery-kid) are Hetzner machines, and delivery-kid's storage is a Hetzner Storage Box. So a copy on another one of our servers protects against losing a disk or a server, but not against losing the Hetzner account. Only a copy outside Hetzner protects against that.[unverified]
At a glance
| What | Lives on | Copies on the same box | Off the box (Hetzner) | Off Hetzner | Checked by |
|---|---|---|---|---|---|
magenta database (magenta_memory: the record, Moods, memory) |
maybelle, magenta-postgres |
2-hourly dumps, 12 h; daily dumps, 7 days | none | NearlyFreeSpeech: a mirror of the 7 daily dumps | cron-health, three stages |
| PickiPedia database (MariaDB) | PickiPedia server | daily dump, 14 days | maybelle (3 days); hunter (14 daily, 8 weekly, 12 monthly) | none | cron-health checks maybelle's copy only |
| PickiPedia files (uploads) | PickiPedia server | none | maybelle (files changed in the last 7 days, kept 14); hunter (accumulates) | none | nothing |
| Release media (IPFS) | delivery-kid, blocks on the Storage Box | none | none | Pinata, when the key is set; unverified | storage audit counts local pins, not Pinata |
| Code | GitHub | n/a | githouse on maybelle (fetches every minute); everyone's clones | GitHub itself | n/a |
| Unpushed work on hunter (branches on no remote, uncommitted edits) | hunter | hourly bundle in /opt/magenta/vcs-backup/ |
maybelle, latest + 30 daily | none | nothing |
Cold start: everything it needs
What the cold start drill needs if every Hetzner machine is gone at once, and where each thing comes back from. This is the drill's step one, the property map. Checked by re at block 26,401,540 (12 Nov 2026) against maybelle-config production @ 38730290. A row without a copy outside Hetzner is a gap.[unverified]
| Needed | Lives on | Copy outside Hetzner | Status |
|---|---|---|---|
| The vault password | people only. maybelle has it only while a deploy runs (see below) | people | who holds it besides Justin is not written anywhere |
| maybelle-config, with the encrypted vault (every secret not made on the box) | githouse on maybelle | GitHub (githouse forwards every push and merge) | ✓ |
| Every other repository (magenta, pickipedia, arthel, rabbithole, pickipedia-mcp …) | githouse | GitHub | ✓ |
githouse's signed merge log (/srv/git/.merges) |
maybelle | none | ✗ small; the merges themselves are on GitHub |
| magenta database | maybelle | NearlyFreeSpeech, 7 days | ✓, but only 7 days deep |
| PickiPedia database | PickiPedia server | none | ✗ |
| PickiPedia files (uploads) | PickiPedia server | none | ✗ |
| PickiPedia's code and settings | pickipedia repo; LocalSettings from the vault | GitHub | ✓ |
| Release media (IPFS blocks) | delivery-kid's Storage Box | Pinata, if every pin reached it | unverified |
| delivery-kid's staging (unpublished uploads, 171 MB at 8 Nov's audit) and seeding directories | Storage Box | none | ✗ unpublished uploads; seeding can be rebuilt from IPFS |
IPFS node identity (/var/lib/ipfs) |
delivery-kid | none | a new node gets a new peer ID; CIDs don't change |
| Unpushed work on hunter | hunter, pulled to maybelle | none | ✗ the drill counts homes as disposable |
| GlitchTip's database | maybelle | none | ✗ error history only |
| justinholmes.com (arthel) | NearlyFreeSpeech | it's built from GitHub | ✓ |
| DNS | cryptograss.live and justinholmes.com on NearlyFreeSpeech; pickipedia.xyz on Namecheap | the providers themselves | records are not in code: new servers mean new addresses, typed in by hand |
| Keys made on the box (githouse's mirror key, the containers' githouse keys, maybelle's backup key) | made by the playbooks | none needed | ✓ made again on deploy; the public halves flow to githouse by themselves |
The vault password
It is not kept on maybelle. maybelle-chapter-1.sh copies it from the deploying person's laptop to /mnt/persist/.vault-password, and the playbook's task "Remove vault password file" deletes it near the end. The six newest maybelle deploy logs (11–12 Nov) list that task as changed, so it was there during the deploy and removed after. The gap: a deploy that failed before that task left the password on the volume until the next one got that far. maybelle-config:re/vault-password-always-removed removes it when the deploy script exits, however the deploy ends. On the same branch, a cron on maybelle (vault-password-sweep.sh, every 10 minutes) removes it if it's there outside a deploy, and cron-health's "vault password sweep" stage then stays red for a day, which posts to #ops. The other targets (hunter, PickiPedia, delivery-kid) write it to /tmp/vault_pass_$$ and remove it on exit already. No backup copies the whole volume, so it is in no backup.[unverified]
magenta database
Still called memory-lane in the code and on the servers. Postgres in the magenta-postgres container on maybelle; data, dumps and daily copies are all on maybelle's one volume, /mnt/persist.[unverified]
- Every two hours, at odd hours,
/mnt/persist/magenta/backup-postgres.shwritesbackups/magenta_memory_<block height>.dump(pg_dump -Fc, about 1.2 GB). It skips the dump if free space is under twice the last one, and deletes any dump that doesn't read back to the end withpg_restore -f /dev/null. Kept 12 hours.[unverified] - Daily: the 03:00 dump is copied to
backups/daily/and kept 7 days.[unverified] - Off the box, and off Hetzner: at 04:00
/usr/local/bin/backup-to-nfs.shrsyncsdaily/to NearlyFreeSpeech (/home/private/backups/) with--delete. That copy is a mirror: the same 7 days, never older. Log:/var/log/nfs-backup.log.[unverified] - Checked by cron-health every 5 minutes: newest dump under 3 h old, newest daily under 26 h, last offsite sync under 26 h old and not empty. At build 56528: dump 0 h old, daily 22 h old (9 kept), offsite sync 21 h ago, 10.4 GB.[unverified]
- Restore: when maybelle's ansible finds the database empty, it lists the dumps in
/mnt/persist/magenta/backups/and asks which one to restore (maybelle/ansible/maybelle.yml). See Cryptograss:Maybelle/Bot for the rebuild.[unverified]
PickiPedia database
MariaDB on the PickiPedia server.[unverified]
- On the box: at 03:00
/usr/local/bin/backup-pickipedia-db.sh(rolepickipedia-vps/ansible/roles/mariadb) writes/var/backups/pickipedia/pickipedia_YYYYMMDD.sql.gzwith--single-transaction, kept 14 days. Same disk as the database. Nothing checks it.[unverified] - Off the box, to maybelle: at 03:30
/usr/local/bin/backup-pickipedia.sh(maybelle/scripts/backup-pickipedia.sh) runsmysqldumpon the server over SSH and writes/mnt/persist/pickipedia/backups/pickipedia_YYYYMMDD.sql.gz(about 233 MB), kept 3 days. Log:/var/log/pickipedia-backup.log.[unverified] - Off the box, to hunter: the same script rsyncs that directory to hunter's
/opt/magenta/pickipedia-backups/, without--delete. hunter'spickipedia-backup-retentiontimer (prune-backups.py) keeps 14 daily, 8 weekly and 12 monthly. Preview environments on hunter load from here (pickipedia#107).[unverified] - Off Hetzner: none.[unverified]
- Checked by cron-health's "pickipedia backup" stage: maybelle's last success under 25 h, newest file over 1,000 bytes. At build 56528: 22 h old, 233,170,403 bytes. The server's own dump and hunter's copy are not checked.[unverified]
- Restore:
maybelle/scripts/import-pickipedia.shandimport-pickipedia-remote.py. Not read for this page.[unverified]
PickiPedia files
Uploads live in /var/www/pickipedia/images on the PickiPedia server: 203 files, 1.66 GB at block 26,374,114 (current versions, from the API).[unverified]
- Off the box: after the database dump, maybelle copies files changed in the last 7 days (leaving out
temp/) into/mnt/persist/pickipedia/backups/images/, kept 14 days, and the rsync to hunter carries them along. Hunter never deletes, so it accumulates: it holds 201 of the 203 current files (missing:Attatchment.XX3c7r.JPEG,IMG_6401_2.JPG). Older files are in hunter's top-level hash directories, newer ones undervar/www/pickipedia/images/.[unverified] - Off Hetzner: none.[unverified]
- Checked by: nothing. A failed copy writes a WARNING to the log, and cron-health doesn't read it.[unverified]
Release media (IPFS)
delivery-kid's kubo keeps its blocks on a Hetzner Storage Box mounted at /mnt/storage-box. When a release is pinned, the pinning service also pins its CID to Pinata if PINATA_JWT is set (delivery-kid/pinning-service/app/services/ipfs.py); a Pinata failure doesn't fail the pin. The storage audit (pages under Cryptograss:Delivery-kid-audits/, newest Cryptograss:Delivery-kid-audits/26371360) compares local pins with Release pages; it doesn't check Pinata. Whether every release has a Pinata copy is unverified.[unverified]
Code
Repositories live on GitHub. githouse on maybelle (/mnt/persist/githouse) fetches GitHub's branches every minute and forwards emergents' pushes to GitHub, so code has at least two copies on different providers, plus everyone's clones.[unverified]
Unpushed work on hunter
Every hour, hunter's hunter-vcs-unpushed writes bundles of branches that exist on no remote, and uncommitted edits, from every home, to /opt/magenta/vcs-backup/. At :20 each hour maybelle's pull-hunter-vcs.sh rsyncs that to /mnt/persist/hunter-vcs/latest/ and keeps one copy a day for 30 days. Nothing checks that the pull succeeds. See Cryptograss:Hunter and Maybelle: Access and Authority.[unverified]
Not backed up
- Everything else in hunter's homes: files outside git, including emergents' local memory directories.[unverified]
- GlitchTip's database (
/mnt/persist/glitchtip/postgres-dataon maybelle).[unverified] - Jenkins build history (jobs are rebuilt from the repo).[unverified]
Gaps, and the fixes we want
- PickiPedia never leaves Hetzner. Database and files both, and the other ✗ rows under Cold start. Fix: one offsite job, to a store outside Hetzner, for everything the cold start needs.[unverified]
- magenta's offsite copy is only 7 days deep. A problem nobody notices for 8 days is in every copy. Fix: the offsite job keeps a history (7 daily, 4 weekly, 12 monthly), and maybelle's key to the store can add but not delete, so whoever controls maybelle can't empty it either.[unverified]
- Copies nobody checks: the PickiPedia server's own dump, hunter's PickiPedia copy, the file copy, the VCS pull, Pinata. Fix: a cron-health stage for each.[unverified]
- hunter's PickiPedia history starts at 4 Oct 2026 (19 dumps), though its retention keeps a year. The pruning code is right, so something else removed the older dumps. Unknown what; hunter's journal for
pickipedia-backup-retentionis the place to look.[unverified] - maybelle's
mysqldumpruns without--single-transaction, so the wiki's tables are locked while it dumps (daily, 03:30). The server's own dump already uses the flag.[unverified] - No restore test. magenta's dumps are read back, which proves they're readable, not that they restore. hunter's previews loading PickiPedia dumps is the nearest thing to a test.[unverified]
Where the offsite copy goes
Prices from each provider's own pricing page, 12 Nov 2026:[unverified]
| Store | Price | Outside Hetzner? | Notes |
|---|---|---|---|
| NearlyFreeSpeech (today's) | $1.00 per GiB-month (site storage) | yes | magenta's 10.4 GB costs about $10 a month; 30 GB would be about $30 |
| Backblaze B2 | $6.95 per TB-month (about $0.007 per GB) | yes | pay for what's stored; object lock and keys that can't delete |
| rsync.net | 1.5¢ per GB-month, 800 GB minimum (about $12 a month) | yes | plain SSH and rsync; snapshots |
| Hetzner Storage Box BX11 | €3.20 a month for 1 TB | no | cheapest per TB, but on the same provider |
re's choice: Backblaze B2, with restic: encrypted before it leaves, deduplicated, retention by rule. 30 GB is about $0.21 a month. The restic password and the B2 key go in the vault, so the chain is: a person holds the vault password, GitHub holds the vault, the vault opens B2. Justin's call: Cryptograss:Moods/backups/todo.[unverified]
Where the code is
All in maybelle-config, branch production:[unverified]
maybelle/ansible/maybelle.yml:backup-postgres.sh,backup-to-nfs.sh,pull-hunter-vcs.sh, and the cron entries for those andbackup-pickipedia.sh.[unverified]maybelle/scripts/backup-pickipedia.sh[unverified]maybelle/jobs/cron-health.groovy: the checks; a stage that changes colour posts to #ops.[unverified]pickipedia-vps/ansible/roles/mariadb/tasks/main.yml: the server's own dump.[unverified]hunter/ansible/roles/housekeeping/:prune-backups.py,vcs-unpushed.py.[unverified]delivery-kid/pinning-service/app/services/ipfs.py: Pinata.[unverified]