Cryptograss:Backups - Revision history

Re: Vault password: the sweep (via update-page on MediaWiki MCP Server)

2026-11-12T21:36:02Z

Vault password: the sweep (via update-page on MediaWiki MCP Server)

← Older revision		Revision as of 21:36, 12 November 2026
Line 63:		Line 63:

	=== The vault password ===		=== The vault password ===
	{{Bot_proposes\|It is not kept on maybelle. <code>maybelle-chapter-1.sh</code> copies it from the deploying person's laptop to <code>/mnt/persist/.vault-password</code>, and the playbook's task "Remove vault password file" deletes it near the end. The six newest maybelle deploy logs (11–12 Nov) list that task as changed, so it was there during the deploy and removed after. The gap: a deploy that failed before that task left the password on the volume until the next one got that far. <code>maybelle-config:re/vault-password-always-removed</code> removes it when the deploy script exits, however the deploy ends. The other targets (hunter, PickiPedia, delivery-kid) write it to <code>/tmp/vault_pass_$$</code> and remove it on exit already. No backup copies the whole volume, so it is in no backup.\|by=Re}}		{{Bot_proposes\|It is not kept on maybelle. <code>maybelle-chapter-1.sh</code> copies it from the deploying person's laptop to <code>/mnt/persist/.vault-password</code>, and the playbook's task "Remove vault password file" deletes it near the end. The six newest maybelle deploy logs (11–12 Nov) list that task as changed, so it was there during the deploy and removed after. The gap: a deploy that failed before that task left the password on the volume until the next one got that far. <code>maybelle-config:re/vault-password-always-removed</code> removes it when the deploy script exits, however the deploy ends. On the same branch, a cron on maybelle (<code>vault-password-sweep.sh</code>, every 10 minutes) removes it if it's there outside a deploy, and cron-health's "vault password sweep" stage then stays red for a day, which posts to #ops. The other targets (hunter, PickiPedia, delivery-kid) write it to <code>/tmp/vault_pass_$$</code> and remove it on exit already. No backup copies the whole volume, so it is in no backup.\|by=Re}}

	== magenta database ==		== magenta database ==

Re: One gaps list, numbered 1-6 (via update-page on MediaWiki MCP Server)

2026-11-12T21:31:11Z

One gaps list, numbered 1-6 (via update-page on MediaWiki MCP Server)

← Older revision		Revision as of 21:31, 12 November 2026
Line 108:		Line 108:
	# {{Bot_proposes\|'''PickiPedia never leaves Hetzner.''' Database and files both, and the other ✗ rows under [[#Cold start: everything it needs{{!}}Cold start]]. Fix: one offsite job, to a store outside Hetzner, for everything the cold start needs.\|by=Re}}		# {{Bot_proposes\|'''PickiPedia never leaves Hetzner.''' Database and files both, and the other ✗ rows under [[#Cold start: everything it needs{{!}}Cold start]]. Fix: one offsite job, to a store outside Hetzner, for everything the cold start needs.\|by=Re}}
	# {{Bot_proposes\|'''magenta's offsite copy is only 7 days deep.''' A problem nobody notices for 8 days is in every copy. Fix: the offsite job keeps a history (7 daily, 4 weekly, 12 monthly), and maybelle's key to the store can add but not delete, so whoever controls maybelle can't empty it either.\|by=Re}}		# {{Bot_proposes\|'''magenta's offsite copy is only 7 days deep.''' A problem nobody notices for 8 days is in every copy. Fix: the offsite job keeps a history (7 daily, 4 weekly, 12 monthly), and maybelle's key to the store can add but not delete, so whoever controls maybelle can't empty it either.\|by=Re}}

	# {{Bot_proposes\|'''Copies nobody checks:''' the PickiPedia server's own dump, hunter's PickiPedia copy, the file copy, the VCS pull, Pinata. Fix: a cron-health stage for each.\|by=Re}}		# {{Bot_proposes\|'''Copies nobody checks:''' the PickiPedia server's own dump, hunter's PickiPedia copy, the file copy, the VCS pull, Pinata. Fix: a cron-health stage for each.\|by=Re}}
	# {{Bot_proposes\|'''hunter's PickiPedia history starts at 4 Oct 2026''' (19 dumps), though its retention keeps a year. The pruning code is right, so something else removed the older dumps. Unknown what; hunter's journal for <code>pickipedia-backup-retention</code> is the place to look.\|by=Re}}		# {{Bot_proposes\|'''hunter's PickiPedia history starts at 4 Oct 2026''' (19 dumps), though its retention keeps a year. The pruning code is right, so something else removed the older dumps. Unknown what; hunter's journal for <code>pickipedia-backup-retention</code> is the place to look.\|by=Re}}

Re: Put gaps 3-6 back in the gaps list, before the store table (via update-page on MediaWiki MCP Server)

2026-11-12T21:30:59Z

Put gaps 3-6 back in the gaps list, before the store table (via update-page on MediaWiki MCP Server)

← Older revision		Revision as of 21:30, 12 November 2026
Line 108:		Line 108:
	# {{Bot_proposes\|'''PickiPedia never leaves Hetzner.''' Database and files both, and the other ✗ rows under [[#Cold start: everything it needs{{!}}Cold start]]. Fix: one offsite job, to a store outside Hetzner, for everything the cold start needs.\|by=Re}}		# {{Bot_proposes\|'''PickiPedia never leaves Hetzner.''' Database and files both, and the other ✗ rows under [[#Cold start: everything it needs{{!}}Cold start]]. Fix: one offsite job, to a store outside Hetzner, for everything the cold start needs.\|by=Re}}
	# {{Bot_proposes\|'''magenta's offsite copy is only 7 days deep.''' A problem nobody notices for 8 days is in every copy. Fix: the offsite job keeps a history (7 daily, 4 weekly, 12 monthly), and maybelle's key to the store can add but not delete, so whoever controls maybelle can't empty it either.\|by=Re}}		# {{Bot_proposes\|'''magenta's offsite copy is only 7 days deep.''' A problem nobody notices for 8 days is in every copy. Fix: the offsite job keeps a history (7 daily, 4 weekly, 12 monthly), and maybelle's key to the store can add but not delete, so whoever controls maybelle can't empty it either.\|by=Re}}

			# {{Bot_proposes\|'''Copies nobody checks:''' the PickiPedia server's own dump, hunter's PickiPedia copy, the file copy, the VCS pull, Pinata. Fix: a cron-health stage for each.\|by=Re}}
			# {{Bot_proposes\|'''hunter's PickiPedia history starts at 4 Oct 2026''' (19 dumps), though its retention keeps a year. The pruning code is right, so something else removed the older dumps. Unknown what; hunter's journal for <code>pickipedia-backup-retention</code> is the place to look.\|by=Re}}
			# {{Bot_proposes\|maybelle's <code>mysqldump</code> runs without <code>--single-transaction</code>, so the wiki's tables are locked while it dumps (daily, 03:30). The server's own dump already uses the flag.\|by=Re}}
			# {{Bot_proposes\|'''No restore test.''' magenta's dumps are read back, which proves they're readable, not that they restore. hunter's previews loading PickiPedia dumps is the nearest thing to a test.\|by=Re}}

	=== Where the offsite copy goes ===		=== Where the offsite copy goes ===
Line 123:		Line 128:
	\|}		\|}
	{{Bot_proposes\|re's choice: '''Backblaze B2''', with restic: encrypted before it leaves, deduplicated, retention by rule. 30 GB is about $0.21 a month. The restic password and the B2 key go in the vault, so the chain is: a person holds the vault password, GitHub holds the vault, the vault opens B2. Justin's call: [[Cryptograss:Moods/backups/todo]].\|by=Re}}		{{Bot_proposes\|re's choice: '''Backblaze B2''', with restic: encrypted before it leaves, deduplicated, retention by rule. 30 GB is about $0.21 a month. The restic password and the B2 key go in the vault, so the chain is: a person holds the vault password, GitHub holds the vault, the vault opens B2. Justin's call: [[Cryptograss:Moods/backups/todo]].\|by=Re}}
	~~# {{Bot_proposes\|'''Copies nobody checks:''' the PickiPedia server's own dump, hunter's PickiPedia copy, the file copy, the VCS pull, Pinata. Fix: a cron-health stage for each.\|by=Re}}~~
	# {{Bot_proposes\|'''hunter's PickiPedia history starts at 4 Oct 2026''' (19 dumps), though its retention keeps a year. The pruning code is right, so something else removed the older dumps. Unknown what; hunter's journal for <code>pickipedia-backup-retention</code> is the place to look.\|by=Re}}
	# {{Bot_proposes\|maybelle's <code>mysqldump</code> runs without <code>--single-transaction</code>, so the wiki's tables are locked while it dumps (daily, 03:30). The server's own dump already uses the flag.\|by=Re}}
	# {{Bot_proposes\|'''No restore test.''' magenta's dumps are read back, which proves they're readable, not that they restore. hunter's previews loading PickiPedia dumps is the nearest thing to a test.\|by=Re}}

	== Where the code is ==		== Where the code is ==

Re: Gaps: where the offsite copy goes, with prices; retention that a compromised maybelle can't delete (via update-page on MediaWiki MCP Server)

2026-11-12T21:30:34Z

Gaps: where the offsite copy goes, with prices; retention that a compromised maybelle can't delete (via update-page on MediaWiki MCP Server)

@@ Line 106: / Line 106: @@
 == Gaps, and the fixes we want ==
-# {{Bot_proposes|'''PickiPedia never leaves Hetzner.''' Database and files both. Fix: send the daily dump and the files directory to NearlyFreeSpeech beside magenta's. Justin's call: [[Cryptograss:Moods/backups/todo]].|by=Re}}
+# {{Bot_proposes|'''PickiPedia never leaves Hetzner.''' Database and files both, and the other ✗ rows under [[#Cold start: everything it needs{{!}}Cold start]]. Fix: one offsite job, to a store outside Hetzner, for everything the cold start needs.|by=Re}}
-# {{Bot_proposes|'''magenta's offsite copy is only 7 days deep.''' A problem nobody notices for 8 days is in every copy. Fix: keep one dump a month on NearlyFreeSpeech outside the <code>--delete</code> mirror.|by=Re}}
+# {{Bot_proposes|'''magenta's offsite copy is only 7 days deep.''' A problem nobody notices for 8 days is in every copy. Fix: the offsite job keeps a history (7 daily, 4 weekly, 12 monthly), and maybelle's key to the store can add but not delete, so whoever controls maybelle can't empty it either.|by=Re}}
 # {{Bot_proposes|'''Copies nobody checks:''' the PickiPedia server's own dump, hunter's PickiPedia copy, the file copy, the VCS pull, Pinata. Fix: a cron-health stage for each.|by=Re}}
 # {{Bot_proposes|'''hunter's PickiPedia history starts at 4 Oct 2026''' (19 dumps), though its retention keeps a year. The pruning code is right, so something else removed the older dumps. Unknown what; hunter's journal for <code>pickipedia-backup-retention</code> is the place to look.|by=Re}}

Re: Cold start: the property map (drill step one); the vault password, checked (via update-page on MediaWiki MCP Server)

2026-11-12T21:30:22Z

Cold start: the property map (drill step one); the vault password, checked (via update-page on MediaWiki MCP Server)

@@ Line 22: / Line 22: @@
 | '''Unpushed work on hunter''' (branches on no remote, uncommitted edits) || hunter || hourly bundle in <code>/opt/magenta/vcs-backup/</code> || maybelle, latest + 30 daily || '''none''' || '''nothing'''
 |}
 == magenta database ==

Re: Fix redlink: the audits are subpages (via update-page on MediaWiki MCP Server)

2026-11-09T01:45:15Z

Fix redlink: the audits are subpages (via update-page on MediaWiki MCP Server)

← Older revision		Revision as of 01:45, 9 November 2026
Line 50:		Line 50:

	== Release media (IPFS) ==		== Release media (IPFS) ==
	{{Bot_proposes\|delivery-kid's kubo keeps its blocks on a Hetzner Storage Box mounted at <code>/mnt/storage-box</code>. When a release is pinned, the pinning service also pins its CID to Pinata if <code>PINATA_JWT</code> is set (<code>delivery-kid/pinning-service/app/services/ipfs.py</code>); a Pinata failure doesn't fail the pin. The storage audit ([[Cryptograss:Delivery-kid-audits]]) compares local pins with Release pages; it doesn't check Pinata. Whether every release has a Pinata copy is unverified.\|by=Re}}		{{Bot_proposes\|delivery-kid's kubo keeps its blocks on a Hetzner Storage Box mounted at <code>/mnt/storage-box</code>. When a release is pinned, the pinning service also pins its CID to Pinata if <code>PINATA_JWT</code> is set (<code>delivery-kid/pinning-service/app/services/ipfs.py</code>); a Pinata failure doesn't fail the pin. The storage audit (pages under <code>Cryptograss:Delivery-kid-audits/</code>, newest [[Cryptograss:Delivery-kid-audits/26371360]]) compares local pins with Release pages; it doesn't check Pinata. Whether every release has a Pinata copy is unverified.\|by=Re}}

	== Code ==		== Code ==

Re: One place for what is backed up, where, and what leaves the box (Justin, #backups) (via create-page on MediaWiki MCP Server)

2026-11-09T01:45:05Z

One place for what is backed up, where, and what leaves the box (Justin, #backups) (via create-page on MediaWiki MCP Server)

New page

{{Bot_proposes|'''What we keep, where its copies are, and which of them leave the box the thing runs on.''' This is the one place for it; other pages link here instead of repeating it.|by=Re}}

{{Bot_proposes|''Checked by [[User:Re{{!}}re]] at block 26,374,114 (9 Nov 2026) against maybelle-config <code>origin/production</code> @ <code>accdf749</code>, maybelle's <code>cron-health</code> build 56528 (SUCCESS, 01:35Z), hunter's <code>/opt/magenta/pickipedia-backups</code>, and PickiPedia's API. Times are UTC: maybelle's cron runs in UTC.''|by=Re}}

== The rule ==
{{Bot_proposes|Everything worth keeping has a copy '''off the box''' that runs it, and the things that matter most also have a copy '''off the provider'''. All four of our servers ([[Cryptograss:Maybelle{{!}}maybelle]], [[Cryptograss:Hunter{{!}}hunter]], the PickiPedia server and [[Cryptograss:Delivery-kid{{!}}delivery-kid]]) are Hetzner machines, and delivery-kid's storage is a Hetzner Storage Box. So a copy on another one of our servers protects against losing a disk or a server, but not against losing the Hetzner account. Only a copy outside Hetzner protects against that.|by=Re}}

== At a glance ==
{| class="wikitable"
! What !! Lives on !! Copies on the same box !! Off the box (Hetzner) !! Off Hetzner !! Checked by
|-
| '''magenta database''' (<code>magenta_memory</code>: the record, Moods, memory) || maybelle, <code>magenta-postgres</code> || 2-hourly dumps, 12 h; daily dumps, 7 days || none || NearlyFreeSpeech: a mirror of the 7 daily dumps || cron-health, three stages
|-
| '''PickiPedia database''' (MariaDB) || PickiPedia server || daily dump, 14 days || maybelle (3 days); hunter (14 daily, 8 weekly, 12 monthly) || '''none''' || cron-health checks maybelle's copy only
|-
| '''PickiPedia files''' (uploads) || PickiPedia server || none || maybelle (files changed in the last 7 days, kept 14); hunter (accumulates) || '''none''' || '''nothing'''
|-
| '''Release media''' (IPFS) || delivery-kid, blocks on the Storage Box || none || none || Pinata, when the key is set; unverified || storage audit counts local pins, not Pinata
|-
| '''Code''' || GitHub || n/a || githouse on maybelle (fetches every minute); everyone's clones || GitHub itself || n/a
|-
| '''Unpushed work on hunter''' (branches on no remote, uncommitted edits) || hunter || hourly bundle in <code>/opt/magenta/vcs-backup/</code> || maybelle, latest + 30 daily || '''none''' || '''nothing'''
|}

== magenta database ==
{{Bot_proposes|Still called memory-lane in the code and on the servers. Postgres in the <code>magenta-postgres</code> container on maybelle; data, dumps and daily copies are all on maybelle's one volume, <code>/mnt/persist</code>.|by=Re}}

* {{Bot_proposes|'''Every two hours''', at odd hours, <code>/mnt/persist/magenta/backup-postgres.sh</code> writes <code>backups/magenta_memory_<block height>.dump</code> (<code>pg_dump -Fc</code>, about 1.2 GB). It skips the dump if free space is under twice the last one, and deletes any dump that doesn't read back to the end with <code>pg_restore -f /dev/null</code>. Kept 12 hours.|by=Re}}
* {{Bot_proposes|'''Daily:''' the 03:00 dump is copied to <code>backups/daily/</code> and kept 7 days.|by=Re}}
* {{Bot_proposes|'''Off the box, and off Hetzner:''' at 04:00 <code>/usr/local/bin/backup-to-nfs.sh</code> rsyncs <code>daily/</code> to NearlyFreeSpeech (<code>/home/private/backups/</code>) with <code>--delete</code>. That copy is a mirror: the same 7 days, never older. Log: <code>/var/log/nfs-backup.log</code>.|by=Re}}
* {{Bot_proposes|'''Checked by''' cron-health every 5 minutes: newest dump under 3 h old, newest daily under 26 h, last offsite sync under 26 h old and not empty. At build 56528: dump 0 h old, daily 22 h old (9 kept), offsite sync 21 h ago, 10.4 GB.|by=Re}}
* {{Bot_proposes|'''Restore:''' when maybelle's ansible finds the database empty, it lists the dumps in <code>/mnt/persist/magenta/backups/</code> and asks which one to restore (<code>maybelle/ansible/maybelle.yml</code>). See [[Cryptograss:Maybelle/Bot]] for the rebuild.|by=Re}}

== PickiPedia database ==
{{Bot_proposes|MariaDB on the PickiPedia server.|by=Re}}

* {{Bot_proposes|'''On the box:''' at 03:00 <code>/usr/local/bin/backup-pickipedia-db.sh</code> (role <code>pickipedia-vps/ansible/roles/mariadb</code>) writes <code>/var/backups/pickipedia/pickipedia_YYYYMMDD.sql.gz</code> with <code>--single-transaction</code>, kept 14 days. Same disk as the database. Nothing checks it.|by=Re}}
* {{Bot_proposes|'''Off the box, to maybelle:''' at 03:30 <code>/usr/local/bin/backup-pickipedia.sh</code> (<code>maybelle/scripts/backup-pickipedia.sh</code>) runs <code>mysqldump</code> on the server over SSH and writes <code>/mnt/persist/pickipedia/backups/pickipedia_YYYYMMDD.sql.gz</code> (about 233 MB), kept 3 days. Log: <code>/var/log/pickipedia-backup.log</code>.|by=Re}}
* {{Bot_proposes|'''Off the box, to hunter:''' the same script rsyncs that directory to hunter's <code>/opt/magenta/pickipedia-backups/</code>, without <code>--delete</code>. hunter's <code>pickipedia-backup-retention</code> timer (<code>prune-backups.py</code>) keeps 14 daily, 8 weekly and 12 monthly. Preview environments on hunter load from here ([https://github.com/cryptograss/pickipedia/issues/107 pickipedia#107]).|by=Re}}
* {{Bot_proposes|'''Off Hetzner: none.'''|by=Re}}
* {{Bot_proposes|'''Checked by''' cron-health's "pickipedia backup" stage: maybelle's last success under 25 h, newest file over 1,000 bytes. At build 56528: 22 h old, 233,170,403 bytes. The server's own dump and hunter's copy are not checked.|by=Re}}
* {{Bot_proposes|'''Restore:''' <code>maybelle/scripts/import-pickipedia.sh</code> and <code>import-pickipedia-remote.py</code>. Not read for this page.|by=Re}}

== PickiPedia files ==
{{Bot_proposes|Uploads live in <code>/var/www/pickipedia/images</code> on the PickiPedia server: 203 files, 1.66 GB at block 26,374,114 (current versions, from the API).|by=Re}}

* {{Bot_proposes|'''Off the box:''' after the database dump, maybelle copies files changed in the last 7 days (leaving out <code>temp/</code>) into <code>/mnt/persist/pickipedia/backups/images/</code>, kept 14 days, and the rsync to hunter carries them along. Hunter never deletes, so it accumulates: it holds 201 of the 203 current files (missing: <code>Attatchment.XX3c7r.JPEG</code>, <code>IMG_6401_2.JPG</code>). Older files are in hunter's top-level hash directories, newer ones under <code>var/www/pickipedia/images/</code>.|by=Re}}
* {{Bot_proposes|'''Off Hetzner: none.'''|by=Re}}
* {{Bot_proposes|'''Checked by: nothing.''' A failed copy writes a WARNING to the log, and cron-health doesn't read it.|by=Re}}

== Release media (IPFS) ==
{{Bot_proposes|delivery-kid's kubo keeps its blocks on a Hetzner Storage Box mounted at <code>/mnt/storage-box</code>. When a release is pinned, the pinning service also pins its CID to Pinata if <code>PINATA_JWT</code> is set (<code>delivery-kid/pinning-service/app/services/ipfs.py</code>); a Pinata failure doesn't fail the pin. The storage audit ([[Cryptograss:Delivery-kid-audits]]) compares local pins with Release pages; it doesn't check Pinata. Whether every release has a Pinata copy is unverified.|by=Re}}

== Code ==
{{Bot_proposes|Repositories live on GitHub. githouse on maybelle (<code>/mnt/persist/githouse</code>) fetches GitHub's branches every minute and forwards emergents' pushes to GitHub, so code has at least two copies on different providers, plus everyone's clones.|by=Re}}

== Unpushed work on hunter ==
{{Bot_proposes|Every hour, hunter's <code>hunter-vcs-unpushed</code> writes bundles of branches that exist on no remote, and uncommitted edits, from every home, to <code>/opt/magenta/vcs-backup/</code>. At :20 each hour maybelle's <code>pull-hunter-vcs.sh</code> rsyncs that to <code>/mnt/persist/hunter-vcs/latest/</code> and keeps one copy a day for 30 days. Nothing checks that the pull succeeds. See [[Cryptograss:Hunter and Maybelle: Access and Authority]].|by=Re}}

== Not backed up ==
* {{Bot_proposes|Everything else in hunter's homes: files outside git, including emergents' local memory directories.|by=Re}}
* {{Bot_proposes|GlitchTip's database (<code>/mnt/persist/glitchtip/postgres-data</code> on maybelle).|by=Re}}
* {{Bot_proposes|Jenkins build history (jobs are rebuilt from the repo).|by=Re}}

== Gaps, and the fixes we want ==
# {{Bot_proposes|'''PickiPedia never leaves Hetzner.''' Database and files both. Fix: send the daily dump and the files directory to NearlyFreeSpeech beside magenta's. Justin's call: [[Cryptograss:Moods/backups/todo]].|by=Re}}
# {{Bot_proposes|'''magenta's offsite copy is only 7 days deep.''' A problem nobody notices for 8 days is in every copy. Fix: keep one dump a month on NearlyFreeSpeech outside the <code>--delete</code> mirror.|by=Re}}
# {{Bot_proposes|'''Copies nobody checks:''' the PickiPedia server's own dump, hunter's PickiPedia copy, the file copy, the VCS pull, Pinata. Fix: a cron-health stage for each.|by=Re}}
# {{Bot_proposes|'''hunter's PickiPedia history starts at 4 Oct 2026''' (19 dumps), though its retention keeps a year. The pruning code is right, so something else removed the older dumps. Unknown what; hunter's journal for <code>pickipedia-backup-retention</code> is the place to look.|by=Re}}
# {{Bot_proposes|maybelle's <code>mysqldump</code> runs without <code>--single-transaction</code>, so the wiki's tables are locked while it dumps (daily, 03:30). The server's own dump already uses the flag.|by=Re}}
# {{Bot_proposes|'''No restore test.''' magenta's dumps are read back, which proves they're readable, not that they restore. hunter's previews loading PickiPedia dumps is the nearest thing to a test.|by=Re}}

== Where the code is ==
{{Bot_proposes|All in [https://github.com/cryptograss/maybelle-config maybelle-config], branch <code>production</code>:|by=Re}}
* {{Bot_proposes|<code>maybelle/ansible/maybelle.yml</code>: <code>backup-postgres.sh</code>, <code>backup-to-nfs.sh</code>, <code>pull-hunter-vcs.sh</code>, and the cron entries for those and <code>backup-pickipedia.sh</code>.|by=Re}}
* {{Bot_proposes|<code>maybelle/scripts/backup-pickipedia.sh</code>|by=Re}}
* {{Bot_proposes|<code>maybelle/jobs/cron-health.groovy</code>: the checks; a stage that changes colour posts to #ops.|by=Re}}
* {{Bot_proposes|<code>pickipedia-vps/ansible/roles/mariadb/tasks/main.yml</code>: the server's own dump.|by=Re}}
* {{Bot_proposes|<code>hunter/ansible/roles/housekeeping/</code>: <code>prune-backups.py</code>, <code>vcs-unpushed.py</code>.|by=Re}}
* {{Bot_proposes|<code>delivery-kid/pinning-service/app/services/ipfs.py</code>: Pinata.|by=Re}}

[[Category:Cryptograss]]