VersionControl:PRs/maybelle-config/250

maybelle-config#250: githouse: our canonical copies on maybelle, mirrored on hunter (ramp Phase 1)

State	🟣 merged 2026-11-06 05:01:31
Worked on by	magent, re, justin
Moods	#ramp
Deploy	redeploy maybelle, then hunter
Branch	magent/githouse → production, by magent-cryptograss
Opened · last pushed	2026-11-01 01:35:15 · 2026-11-06 04:50:13

githouse: our canonical copies of the repositories, on maybelle, with a read-only mirror on hunter. The ramp's Phase 1 ([[Cryptograss:Moods/ramp/Plan]], section 3), as Justin, re and magent settled it in #ramp between 31 Oct and 6 Nov. Reworked from the first version (hunter-only) after Justin's call on 2 Nov that the copy deploys come from belongs on maybelle, and re's ten points the same day.

**On maybelle** (`shared/ansible/roles/githouse`, included from `maybelle.yml`; the code is `shared/githouse/githouse.py`, one file, tested against real repositories as the image is built):
- One bare repository per repo in `shared/githouse/repos.txt`, at `/mnt/persist/githouse/git`.
- **GitHub's branches** (`main`, `master`, `production`, `release/*`) and pull requests' heads are fetched every 60 s, spread across the minute, nothing fetched when nothing changed, forced and pruned to match GitHub. Over ssh with Jenkins's GitHub key (`jenkins_github_ssh_key`, already in the vault): no new secret.
- **Pushes** go over ssh to `10.0.0.2:2222`, the private network only. The role asserts the bind address is private and refuses `0.0.0.0`, and checks after starting that docker published it on that address alone (re's point 5). Each key's forced command names who it is; that name may push only `<name>/*`, no tags, never a deploy branch. The fetch never touches prefixed branches, and nothing deletes them on its own (re's point 1).
- **A name is used once:** merged branches move to `refs/merged/`, deleted ones to `refs/closed/`, and a push that would create either name again is refused with the next free one. Merged means the tip is in a GitHub branch's history but not on its first-parent line. Needs merge commits; squash, rebase and merge queue break it.
- **Pushes are forwarded to GitHub** with the same key, so pull requests keep working until the flip (re's point 6). A failed forward is logged, never retried by deleting.
- **Host key** lives on the maybelle volume, so a rebuilt container is the same host (re's point 4).
- **Keys** (re's point 1, Justin 1 Nov: not a vault secret): every container on hunter makes its own users' keys and writes the public halves to `/opt/magenta/githouse/keys/<container>/<name>.pub`. maybelle pulls that directory every minute (rsync with root's existing hunter key) and builds `authorized_keys` from it as `githouse_pushers` in `maybelle.yml` allows: `emergent-re` may vouch for `re`, `emergent-blu` for `blu`, `emergent-magent` for `magent`, and nobody else for anybody. A key under another name is ignored; a file gone from the relay is gone from `authorized_keys` at the next minute; nothing is carried forward. hunter's own mirror key is `fetch_only` (re's point 2): hunter can't mint itself a pushable identity. A key is honoured within about two minutes of a container's first start; a push before that gets "Permission denied (publickey)" (re's point 3).
- **`branches.json`** is read by memory-lane from the same volume, mounted read-only, and served at `/api/branches/` (magenta#333). No Caddy host, no DNS (re's point 7).

**On hunter** (`hunter/ansible/roles/githouse-mirror`): the same image in mirror mode, fetching everything from maybelle every minute into `/opt/magenta/git`, mounted read-only at `/srv/git` in every container. Nothing there needs backing up. Each container also mounts its relay directory at `/srv/githouse-keys`.

**In the containers** (`container_startup.py`): each emergent, and magent, gets a key at `~/.config/magenta/githouse_key` (made once; the public half published), and each clone a remote `hunter` that fetches from the mount and pushes to `ssh://git@10.0.0.2:2222/<repo>.git` with that key (`core.sshCommand` per clone, so nothing in `~/.ssh`, which is mounted read-only for magent). `origin` stays GitHub. New clones use `--reference-if-able` against the mirror.

**Not done here, by design:** deploys still clone from GitHub (re's point 9: a Jenkinsfile change, its own PR); the name registry (`refs/merged/`, `refs/closed/`) and the host key are the things a maybelle backup has to carry, and that waits on the backup plan in #emergence. The interim: merges still happen on GitHub and maybelle's `main` lags up to 60 s, so a just-merged name is briefly reusable (re's point 8).

**Checked:** `test_githouse.py`, 16 tests against real git repositories (fetch isolation, push rules, retiring names and the retire race, forwarding and its failure, the relayed-keys builder, the mirror, auth incl. fetch-only). `test_container_startup.py`, 36 (4 new: the remote, the push URL, no ssh target, the key made once and published). The image builds and its tests pass inside the build (`docker build` on hunter). Both playbooks parse (`--syntax-check`), and both roles ran in `--check` mode with the vars rendered, including the bind assertion refusing `0.0.0.0`. **Not run on maybelle or hunter.** The first deploy is the test; maybelle first, then hunter (the mirror needs maybelle to have its key, which arrives through the relay within a minute of hunter's deploy).

Before merging, one thing to confirm, Justin: `jenkins_github_ssh_key` can push to the org's repos. If it's read-only, fetching works and forwarding fails into `githouse.log`; pushes then stay on maybelle until a key that can push replaces it.

**re's review (6 Nov 04:44Z), folded in as `50772be`:** the hooks run under sshd's forced command without entrypoint.sh's exports, and `main()` set `GIT_SSH_COMMAND` only after dispatching them, so every forward to GitHub would have run without the key and failed into the log. Real bug; now set first, with a test. Her second finding, that the port check matched the line's start and would fail every deploy, rested on `docker port`'s no-argument format (`22/tcp -> addr:port`); with the port argument it prints `addr:port` lines, so the old check was right. The new one is still better: it also fails on no lines at all (container not up) and on any extra binding, such as the `[::]` line a `0.0.0.0` bind adds.


Deploy: redeploy maybelle, then hunter
Worked on by: magent, re, Justin, in #ramp