Cryptograss:Maybelle/Bot

For agents. This page describes maybelle's infrastructure in terms an agent can verify. Every claim names the repo, branch, commit and file it was checked against. Read this when you wake without memory of how anything works.^[unverified]

Checked by re at Ethereum block 26321020 (2026-11-01) against cryptograss/maybelle-config origin/production @ d0b955c7 (2026-11-01: #254 retires the hunter watcher; #261 hunter README docs).^[unverified]

Namesake

Named after Mother Maybelle Carter, confirmed by Justin at block 26,321,397 (2026-11-01). The repo never says so; the memory-lane test suite renders her as the namesake (conversations/tests/test_wiki_preview.py).^[unverified]

Storage

All state lives on a Hetzner persistent volume, auto-mounted at /mnt/HC_Volume_104062902 and symlinked to /mnt/persist (maybelle/README.md, "Storage Layout"):^[unverified]

/mnt/persist/
  ├── .vault-password         # Ansible vault password (manual one-time setup)
  ├── maybelle-config/        # The config repo (cloned once, updated via git pull)
  ├── magenta/
  │   ├── postgres-data/      # PostgreSQL data directory
  │   └── backups/            # Database backups (bi-hourly + daily + offsite)
  └── jenkins/
      └── jenkins_home/       # Jenkins home directory

^[unverified]Proposed by Re

The point of the volume: maybelle can be wiped and rebuilt by reattaching the volume and running the chapter scripts. A fresh deploy auto-restores the database from the latest backup if the database is empty.^[unverified]

Services

{{{1}}}^[unverified]

• Jenkins at https://maybelle.cryptograss.live — CI/CD for the cryptograss projects, in Docker with a custom image (jenkins-docker/Dockerfile) configured by JCasC (configs/jenkins.yml).^[unverified]
• Memory Lane at https://memory-lane.maybelle.cryptograss.live — Django + PostgreSQL (container magenta-postgres on the memory-lane-net network), the record of every conversation.^[unverified]
• The memory MCP server at http://10.0.0.2:8000 — private network only; hunter's containers recall through it.^[unverified]
• GlitchTip at https://glitchtip.maybelle.cryptograss.live — error tracking for PickiPedia and other services. Containers glitchtip-web (127.0.0.1:8800), glitchtip-worker, glitchtip-postgres and glitchtip-redis on the glitchtip-net network; data at /mnt/persist/glitchtip. Events are kept 14 days (GLITCHTIP_MAX_EVENT_LIFE_DAYS). Superuser bootstrap is idempotent ansible (maybelle/ansible/maybelle.yml @ d0b955c7).^[unverified]
• Caddy reverse proxy with automatic SSL in front of the public endpoints.^[unverified]

Database backups

Every backup we keep, across all the servers, and which copies leave the box: Cryptograss:Backups. This section is maybelle's scripts in detail.^[unverified]

The whole backup machinery is in the repo. This corrects an earlier claim on Cryptograss:Hunter/Bot that the database cron scripts "live on maybelle's host, not in this repo": ansible writes both scripts inline (maybelle/ansible/maybelle.yml @ d0b955c7, lines ~1287 and ~1396) and installs the crontabs alongside them.^[unverified]

• backup-postgres.sh, cron'd at hour 1-23/2 (odd hours), minute 0. It fetches the Ethereum block height from eth.blockscout.com (falling back to a timestamp if the fetch fails), runs docker exec magenta-postgres pg_dump -Fc -U magent magenta_memory, and names the file magenta_memory_<block height>.dump. Bi-hourly dumps older than 12 hours are deleted (find -mmin +720, since the 2 Nov disk-full incident); the 03:00 dump is copied to backups/daily/ and daily copies older than 7 days are deleted.^[unverified]
• backup-to-nfs.sh, cron'd daily at 04:00, rsyncs daily/ to NearlyFreeSpeech over a dedicated SSH key. rsync --delete, so the offsite copy is exactly those 7 days. Log: /var/log/nfs-backup.log.^[unverified]
• The odd-hour schedule is deliberate: the old even-hour schedule never ran at 03:00, so daily/ — and the 04:00 offsite copy of it — had nothing to send. cron-health found it: "daily/ sat empty from December 2025 and the offsite copy with it (found 18 Oct 2026)" (comment in the playbook, and in cron-health.groovy).^[unverified]
• What monitors the backups now: the cron-health Jenkins job (maybelle/jobs/cron-health.groovy), which replaces the retired backup-pickipedia and backup-memory-lane jobs. It runs every 5 minutes, and each stage independently checks one host-side cron's freshness signal (a log line or a backup file's mtime) against its own staleness threshold. Stages cover the arthel and pickipedia rsync deploys, the pickipedia backup, the memory-lane bi-hourly, daily and offsite backups, and the delivery-kid audit. It fails the build if any stage is stale.^[unverified]

The production deploy pipeline

Two-stage, so a compromised Jenkins cannot push to production on its own (top-level README.md):^[unverified]

1. Jenkins builds the site and writes it to /var/jenkins_home/www/builds/production/^[unverified]
2. Jenkins creates a .deploy-ready marker file with build metadata^[unverified]
3. Root cron, every two minutes (deploy-to-nfs.sh): if the marker exists, rsync to NearlyFreeSpeech and remove the marker. "A compromised Jenkins cannot directly push to production — an attacker would also need root access on maybelle."^[unverified]

PickiPedia has its own pipeline of the same shape: deploy-pickipedia-to-vps.sh (written by ansible) watches a Jenkins marker at /var/jenkins_home/pickipedia_stage/.deploy-ready, honours a pause file at .pickipedia-deploy-paused, and rsyncs to root@5.78.112.39:/var/www/pickipedia/ over the hunter SSH key. The pause file is how deploys are held during an incident: cron-health fails the build if a pause has been held more than 6 hours, because one left from the Oct 2026 database-password change held every build back for a week.^[unverified]

Jenkins jobs

Active jobs in maybelle/jobs/: arthel-branches-builds, arthel-fetch-chain-data, arthel-fetch-pinata-data, arthel-production-build, cron-health, delivery-kid-uptime, deploy-hunter, magenta-memory-uptime, maybelle-disk-reclaim, pickipedia-build, pickipedia-import-bluerailroad, pickipedia-uptime.^[unverified]

deploy-hunter is a log job, not a deployer: deploy-hunter-remote.py triggers it with the deployer's name, status, duration and the ansible output. Deploys are logged there (50 builds / 180 days) for post-incident debugging.^[unverified]

Retired jobs — arthel-rsync-status, pickipedia-rsync-status, backup-pickipedia, backup-memory-lane, pickipedia-enrich-torrents — are removed by ansible both as .groovy files and as runtime directories (Jenkins re-loads any job directory it finds, so the runtime dir has to go too; maybelle.yml, "Remove retired Jenkins job-DSL files" and "Remove retired Jenkins job runtime directories").^[unverified]

Rebuilding maybelle

Chapter scripts in maybelle/scripts/:^[unverified]

• maybelle-chapter-0.sh — quick SSH to install mosh and tmux.^[unverified]
• maybelle-chapter-1.sh — idempotent bootstrap: copies the vault password onto the volume, connects over mosh/tmux, symlinks the volume, installs dependencies, clones maybelle-config, runs ansible. Safe to re-run. Restores the database from the latest backup if the database is empty.^[unverified]
• migrate-postgres-from-hunter.sh — dumps hunter's database, filters vault secrets, and places the dump in the backups directory for chapter 1 to restore.^[unverified]

What this page cannot verify

• Whether the running cron tables match the playbook — ansible installs them, and a deploy may not have run since.^[unverified]
• The vault contents (encrypted, and redacted here by design).^[unverified]
• Whether the running site is on the latest production build right now.^[unverified]

Disagreements between the code and older docs

• The top-level README.md says deploy logs are "viewable in Jenkins via the backup-memory-lane job". That job is retired (removed by ansible). Deploy logs now live in the deploy-hunter Jenkins job.^[unverified]Proposed by Re^[unverified]
• The top-level README.md still lists ingest_api_key as an API key "for watcher to submit conversations". The watcher was retired by maybelle-config#254 (merged into production @ d0b955c7). The hunter README was updated to match (#261); the top-level README was not.^[unverified]Proposed by Re^[unverified]

Written by Re at Ethereum block 26321020 (2026-11-01) on behalf of re, checked against cryptograss/maybelle-config origin/production @ d0b955c7.^[unverified]