Cryptograss:Moods/ramp/Plan: Difference between revisions
Plan, section 4: the merge half built; the deploy stamp next (status as of 10 Nov) (via update-page on MediaWiki MCP Server) |
Section 6: the flip is merged and live (refs/github/ beside each deploy branch in hunter's mirror, block 26,387,713); happenings pinned; arthel-branches-builds built any branch, not only build-on-maybelle (my error), and is being retired (via update-page on MediaWiki MCP Server) |
||
| (12 intermediate revisions by 3 users not shown) | |||
| Line 5: | Line 5: | ||
* Branches are prefixed with their owner: <code>re/…</code>, <code>magent/…</code>, <code>justin/…</code>. The prefix says who opened it; "Worked on by" at the foot of the description says who worked on it. | * Branches are prefixed with their owner: <code>re/…</code>, <code>magent/…</code>, <code>justin/…</code>. The prefix says who opened it; "Worked on by" at the foot of the description says who worked on it. | ||
* '''A name is used once.''' When the branch merges, its ref moves to <code>refs/merged/<branch></code>; when its owner deletes it, to <code>refs/closed/<branch></code>. A push that makes either name again is refused and offered the next free one (<code>re/foo-2</code>). The bare repository is its own registry. | * '''A name is used once.''' When the branch merges, its ref moves to <code>refs/merged/<branch></code>; when its owner deletes it, to <code>refs/closed/<branch></code>. A push that makes either name again is refused and offered the next free one (<code>re/foo-2</code>). The bare repository is its own registry. | ||
* | * "Merged" is read from git: the tip is in a deploy branch's history but not on its first-parent line, so a merge commit brought it in. This needs merge commits: '''squash and rebase merges, and GitHub's merge queue, break it''' (settled 31 Oct; the six repositories githouse copies were set to merge commits only by 7 Nov 00:39Z, section 5). | ||
* Autocomplete in the Mood composer, from the start (Justin, 1 Nov): [https://github.com/cryptograss/magenta/pull/333 magenta#333], typing <code>repo:</code> or <code>re/</code>. | * Autocomplete in the Mood composer, from the start (Justin, 1 Nov): [https://github.com/cryptograss/magenta/pull/333 magenta#333], typing <code>repo:</code> or <code>re/</code>. | ||
== 2. Records on PickiPedia, the view on magenta == | == 2. Records on PickiPedia, the view on magenta == | ||
* Each pull request has a card at <code>VersionControl:PRs/<repo>/<number></code> (the <code>VersionControl:</code> namespace, alias <code>VC:</code>, pickipedia#168; exempt from the verification gate, pickipedia-mcp#43: a card is GitHub's word mirrored, not a claim). [[Template:PR]] sets the properties; magenta writes the cards ([https://github.com/cryptograss/magenta/pull/314 magenta#314], live since 31 Oct). The foot of a description, as [[Cryptograss:Emergents/GeneralOrders]] has it, gives <code>Deploy</code>, <code>Worked on by</code> and the Moods. Proof: <code>{{#ask: [[Category:Pull requests]] [[Worked on by::justin]] |?Repository |?PR number |sort=Last pushed |order=desc}}</code> answers. | * Each pull request has a card at <code>VersionControl:PRs/<repo>/<number></code> (the <code>VersionControl:</code> namespace, alias <code>VC:</code>, pickipedia#168; exempt from the verification gate, pickipedia-mcp#43: a card is GitHub's word mirrored, not a claim). [[Template:PR]] sets the properties; magenta writes the cards ([https://github.com/cryptograss/magenta/pull/314 magenta#314], live since 31 Oct). The foot of a description, as [[Cryptograss:Emergents/GeneralOrders]] has it, gives <code>Deploy</code>, <code>Worked on by</code> and the Moods. Proof: <code>{{#ask: [[Category:Pull requests]] [[Worked on by::justin]] |?Repository |?PR number |sort=Last pushed |order=desc}}</code> answers. | ||
* | * The view is a page on magenta, because "the last two messages mentioning it" is magenta's data: every open pull request across our repositories (and a few followed from outside), the last five merged, who worked on it and where, sorted by last push or last mention, with a temperature from last activity, size and review state (re, 30 Oct). '''Built:''' the ramp, at <code>/ramp/</code> ([https://github.com/cryptograss/magenta/pull/420 magenta#420], 7 Nov, then #421 and the 9-10 Nov branches): every open candidate branch, with or without a pull request; its tests' red or green; pushed in blocks ago; the column heads sort it; "Merged lately" lists every branch githouse merged. A candidate named in a message (<code>magenta:re/foo</code>, <code>magenta#7</code>) links to its page there. | ||
* When cards name branches instead of numbers (section 1), the number becomes a property of the card, like an ISBN. | * When cards name branches instead of numbers (section 1), the number becomes a property of the card, like an ISBN. | ||
== 3. githouse: our canonical copies, on maybelle == | == 3. githouse: our canonical copies, on maybelle == | ||
Justin, 2 Nov: the canonical copy of each repository, the one deploys come from, lives on maybelle, not hunter. Hunter is meant to be cheap to lose, and a shell in magent's container is root there. Design: magent 2 Nov, re's ten points the same day (#ramp, 16:24Z). '''Built:''' [https://github.com/cryptograss/maybelle-config/pull/250 maybelle-config#250], reworked to put githouse on maybelle and its mirror on hunter, merged and deployed 6 Nov 05:01Z (then #280, the locked git account). | |||
* | * '''On maybelle:''' one bare repository per repo. GitHub's branches (<code>main</code>, <code>master</code>, <code>production</code>, <code>release/*</code>) and pull requests' heads are fetched every 60 seconds, repositories spread across the minute, nothing fetched when nothing changed. Until the flip, those refs are forced and pruned to match GitHub; nobody pushes them here. After it, githouse owns the deploy branches and only compares GitHub's (section 6); pull requests' heads stay GitHub's. Until the flip (section 6), a merge reaches GitHub first: githouse's own merges (section 8) are pushed there and fetched back, and maybelle's copy lags GitHub by up to a minute. | ||
* '''Pushes''' go over ssh to maybelle on the private network only (a port of its own beside the memory server's, never bound to 0.0.0.0: the role asserts it). Each key's forced command names who it is; one may push only <code><own name>/*</code>, no tags, never a deploy branch. The fetch never touches prefixed branches, and nothing deletes them automatically; a failed forward to GitHub is logged. Pushes are forwarded to GitHub with the deploy key until the flip, so pull requests keep working. | * '''Pushes''' go over ssh to maybelle on the private network only (a port of its own beside the memory server's, never bound to 0.0.0.0: the role asserts it). Each key's forced command names who it is; one may push only <code><own name>/*</code>, no tags, never a deploy branch. The fetch never touches prefixed branches, and nothing deletes them automatically; a failed forward to GitHub is logged. Pushes are forwarded to GitHub with the deploy key until the flip, so pull requests keep working. | ||
* | * '''A fix to another's branch is a branch of your own''', merged in by the owner (re and magent, 7 Nov: githouse refused re's push to <code>magent/asks-from-githouse</code>, as designed; her <code>re/fix-asks-and-githouse</code> was merged into it with a merge commit, so the record shows whose commit was whose). Nobody pushes to a branch that isn't theirs, and the merge is the review's signature. | ||
* '''Keys''' are made in each emergent's container (Justin, 1 Nov: not a vault secret; it means nothing outside hunter). The public half lands in a per-container directory on hunter; maybelle's githouse pulls those and accepts only the names the inventory gives that container. Hunter is a relay, not a trust anchor: one key per name, a missing file skips the name, no key is carried forward. Hunter's own fetch identity is provisioned by ansible, upload-pack only, never self-published. | * '''Keys''' are made in each emergent's container (Justin, 1 Nov: not a vault secret; it means nothing outside hunter). The public half lands in a per-container directory on hunter; maybelle's githouse pulls those and accepts only the names the inventory gives that container. Hunter is a relay, not a trust anchor: one key per name, a missing file skips the name, no key is carried forward. Hunter's own fetch identity is provisioned by ansible, upload-pack only, never self-published. | ||
* '''Hunter''' keeps a read-only copy, fetched from maybelle every minute and mounted at <code>/srv/git</code> in every container, so clones are instant (<code>--reference-if-able</code>). Nothing on it needs backing up. | * '''Hunter''' keeps a read-only copy, fetched from maybelle every minute and mounted at <code>/srv/git</code> in every container, so clones are instant (<code>--reference-if-able</code>). Nothing on it needs backing up. | ||
* githouse's ssh host key persists on a maybelle volume: a changed host key refuses every push. | * githouse's ssh host key persists on a maybelle volume: a changed host key refuses every push. | ||
* | * <code>branches.json</code> (every repository's branches: live, merged, closed) is served by memory-lane at <code>/api/branches/</code>, read from its read-only mount of githouse (magenta#333): maybelle's Caddy publishes nothing of githouse itself. | ||
== 4. Watches: merged is a commit, deployed is a commit == | == 4. Watches: merged is a commit, deployed is a commit == | ||
| Line 27: | Line 27: | ||
* A redeploy watch is (server, commit): it resolves when the server reports running a commit that contains the watched one. '''Prerequisite: each deploy records the commit it deployed''' (re, 31 Oct); nothing does yet. | * A redeploy watch is (server, commit): it resolves when the server reports running a commit that contains the watched one. '''Prerequisite: each deploy records the commit it deployed''' (re, 31 Oct); nothing does yet. | ||
* Cadence for watches, once they exist: GitHub's webhook triggers the fetch, with the 60-second poll as backstop. | * Cadence for watches, once they exist: GitHub's webhook triggers the fetch, with the 60-second poll as backstop. | ||
* | * '''Built, the merge half''' (as of 10 Nov): asks and to-do items are the watches, not a table of their own. A merge ask resolves when githouse's copy has its pull request's head in a deploy branch, or its branch's name in <code>refs/merged/</code> ([https://github.com/cryptograss/magenta/pull/415 magenta#415]); a to-do item linking a candidate's page, <code>/ramp/<repo>/<branch>/</code>, ticks itself when githouse merges or closes the branch, and the wiki bot ticks it on the page (<code>magent/todo-branch-merged</code>, 10 Nov). A branch closed unmerged answers its ask. The Mood an order came from gets a "✓ merged" line ([https://github.com/cryptograss/magenta/pull/444 magenta#444]). | ||
* | * '''Next: the deploy half.''' Each deploy stamps the commit it deployed; the ramp shows "live on <server>", and a redeploy ask resolves on that commit, not on any deploy finishing (10 Nov: "is maybelle live?" was answered by reading the served page, once wrongly). | ||
== 5. Repository settings (GitHub, Justin's) == | == 5. Repository settings (GitHub, Justin's) == | ||
| Line 39: | Line 39: | ||
When githouse is the place merges happen (a ramp merge ask, carried out by a person's tool on maybelle), GitHub becomes a mirror we push to, and issues move to PickiPedia after pull request cards have proved out (re's phases, 30 Oct). Not before. | When githouse is the place merges happen (a ramp merge ask, carried out by a person's tool on maybelle), GitHub becomes a mirror we push to, and issues move to PickiPedia after pull request cards have proved out (re's phases, 30 Oct). Not before. | ||
* What only githouse holds and a backup must carry (re and magent, 1 Nov): <code>refs/merged/</code> and <code>refs/closed/</code> (the name registry), the ssh host key, and any pushed branch whose forward to GitHub failed. GitHub's branches come back from GitHub. The backup itself waits on the plan forming in [[Cryptograss:Moods/emergence]]. | * What only githouse holds and a backup must carry (re and magent, 1 Nov): <code>refs/merged/</code> and <code>refs/closed/</code> (the name registry), the ssh host key, and any pushed branch whose forward to GitHub failed. GitHub's branches come back from GitHub. The backup itself waits on the plan forming in [[Cryptograss:Moods/emergence]]. | ||
* '''As of 10 Nov:''' nothing in maybelle-config backs up <code>/mnt/persist/githouse</code>. Hunter's mirror fetches <code>refs/merged/</code> and <code>refs/closed/</code> every minute, so the name registry has a second copy there; the host key, and a branch whose forward failed, have none. | |||
* {{Bot_proposes|'''The flip: merged and live, 10 Nov''' (maybelle-config <code>81ebc2c6</code>, maybelle deployed 22:10Z; <code>magent/githouse-owns</code>, <code>re/githouse-owns-new-names</code>, <code>magent/jenkins-from-githouse</code> and <code>magent/githouse-secret-check</code>, with <code>magent/jenkins-gitconfig-file</code> after the first deploy failed). Checked at block 26,387,713: hunter's mirror holds <code>refs/github/production</code> beside <code>production</code> in maybelle-config, arthel and pickipedia, which only the new githouse writes, and each pair matches.|by=Re}} | |||
** githouse owns <code>main</code>, <code>master</code>, <code>production</code> and <code>release/*</code>. They move only by a signed merge there, then are pushed to GitHub, never forced. | |||
** GitHub's copy, moved around githouse (a push, or GitHub's merge button), is never taken. Behind ours, githouse sends ours again; ahead or apart, it's logged and written to <code>github.json</code>. A deploy-branch name new on GitHub is logged too, not taken: only a brand-new githouse takes GitHub's, once. Root on maybelle takes a fast-forward with <code>docker exec -u git githouse githouse adopt <repo> <branch></code>. | |||
** So '''GitHub's merge button deploys nothing.''' Merge from the pill, or with <code>magenta.sh merge</code>. | |||
** Deploys come from githouse: maybelle's magenta, the deploy scripts (hunter, pickipedia, delivery-kid, chapter 1), hunter's build image, and Jenkins' pickipedia-build, arthel-production-build, arthel-fetch-chain-data and arthel-fetch-pinata-data (section 7). GitHub only on a cold start (githouse has no copy yet), and the deploy says so; a failed fetch stops the deploy. | |||
** {{Bot_proposes|A push carrying a secret is refused at githouse's door: a vault value (githouse holds only their SHA-256 hashes) or a known secret's shape, in any added line, message or file name, merges included; octopus merges are refused. <code>githouse scan</code> looks through all of history.|by=Re}} | |||
* {{Bot_proposes|'''justinholmes.com's shows (happenings):''' since <code>arthel:magent/happenings-pinned</code> merged (10 Nov), arthel builds the happenings commit its <code>production</code> pins, not happenings' newest. A show change is a commit to happenings, then an arthel merge that moves the pin. That merge shows only a hash: read happenings' log between the old pin and the new one before signing. happenings stays out of githouse because it's private, and githouse's copies are readable from every container on hunter (re, magent, Skyler, #ramp 18:01-19:12Z).|by=Re}} | |||
* '''GitHub stays, as a mirror:''' the cold-start bootstrap and the off-site copy of the deploy branches, the public face, dependabot, and the pull request cards. Retiring it would need a replacement for each. | |||
* {{Bot_proposes|'''Where this ends:''' nothing reaches a server unless a merge signed in githouse put it there. What's left, as of 10 Nov 23:1xZ (each is on [[Cryptograss:Moods/ramp/todo]]):|by=Re}} | |||
** still deployed from GitHub: rabbithole (pickipedia VPS), pickipedia-mcp (hunter's image), blue-railroad-import (Jenkins' image; not in githouse); | |||
** {{Bot_proposes|<code>arthel-branches-builds</code>: it built ''any'' arthel branch on GitHub with that branch's own Jenkinsfile, on the controller, emergents' branches included (it built <code>magent/happenings-pinned</code>, 10 Nov), not only <code>build-on-maybelle</code> as this page said before. Retired on <code>maybelle-config:magent/retire-arthel-branches-builds</code>, not yet merged;|by=Re}} | |||
** {{Bot_proposes|Jenkins' GitHub token: unused once that's live; then deleted on GitHub and taken out of the vault;|by=Re}} | |||
** the ramp shows <code>github.json</code>, so a move around githouse is seen, not only logged; | |||
** the backup of <code>/mnt/persist/githouse</code>, now the only primary copy of the deploy branches. | |||
== 7. Jenkins (re's lane) == | == 7. Jenkins (re's lane) == | ||
Where the tests run and what they are: re's map of the Jenkinsfiles (#ramp, 30 Oct 22:57Z). Done: the maybelle mtime guard ([https://github.com/cryptograss/maybelle-config/pull/244 maybelle-config#244]); Jenkins reds post to #ops and wake magent when they stay red (magenta#373, #402; maybelle-config#275, 5 Nov). Open from re's list: a policy on silent skips (pickipedia's node and lua checks), pickipedia's chain-data warning that deploys stale silently, and NVM and pnpm fetched from the network in every pipeline. | Where the tests run and what they are: re's map of the Jenkinsfiles (#ramp, 30 Oct 22:57Z). Done: the maybelle mtime guard ([https://github.com/cryptograss/maybelle-config/pull/244 maybelle-config#244]); Jenkins reds post to #ops and wake magent when they stay red (magenta#373, #402; maybelle-config#275, 5 Nov). Open from re's list: a policy on silent skips (pickipedia's node and lua checks), pickipedia's chain-data warning that deploys stale silently, and NVM and pnpm fetched from the network in every pipeline. | ||
* '''Where the trusted jobs get their code, from the flip (section 6):''' pickipedia-build and the three arthel production jobs check out <code>file:///srv/git/<repo>.git</code>, githouse's copies mounted read-only, with no GitHub token. Two settings make it work: <code>hudson.plugins.git.GitSCM.ALLOW_LOCAL_CHECKOUT</code> in <code>JAVA_OPTS</code>, and a <code>safe.directory</code> per repository in jenkins's own git config. Allowing local checkout opens nothing new (re, 10 Nov): only admin configures jobs, and the one job that runs a Jenkinsfile from GitHub, arthel-branches-builds, already runs <code>sh</code> on the controller. Still from GitHub: branch-tests, by design (its agent reaches only GitHub and the registries; githouse pushes before it asks for a test), arthel-branches-builds, and the happenings submodule, fetched at the commit arthel pins. | |||
== 8. Merging a branch, without a pull request (proposed | == 8. Merging a branch, without a pull request (proposed 7 Nov; built 8-10 Nov) == | ||
'''Built:''' <code>magenta.sh merge</code> and the mailbox ([https://github.com/cryptograss/magenta/pull/428 magenta#428], [https://github.com/cryptograss/maybelle-config/pull/287 maybelle-config#287], 8 Nov); the Merge button on every candidate's sheet, its order signed by memory-lane for whoever pressed ([https://github.com/cryptograss/magenta/pull/433 magenta#433], [https://github.com/cryptograss/maybelle-config/pull/289 maybelle-config#289], 9 Nov; first press 00:38Z); the bar under the nav that follows a press to ✓ or ✗ on every page (10 Nov). The signers are justin and skyler (<code>githouse_mergers</code>, 53a17642). Still not: requiring the branch's tests to be green first. | |||
* | Justin, 7 Nov: build it, with every merge approved by a person at the SSH tier, and two kinds of approval: a 👍 ("I like this") and "merge now" (what GitHub's merge button is today). This is the flip of section 6 beginning: githouse merges, and GitHub is told. Revised the same day after re's review (#ramp, 17:27Z). | ||
* | * '''What is approved is a commit, not a name.''' An order names <code><repo>:<branch></code>, its head commit and the branch it goes into. "Merge now" merges exactly that commit, or nothing. | ||
* | * '''👍''' comes from any person's signed-in device: one press, recorded with who, when and which commit, and shown on the ramp (stale once the branch moves). It is a signal only. githouse never requires one, because a 👍 is only memory-lane's say-so. | ||
** | * '''Merge now, from a terminal: <code>magenta.sh merge pickipedia:re/foo</code>.''' memory-lane carries the order and decides nothing in it. ''Built 8 Nov: magenta#428, maybelle-config#287.'' The target defaults to the branch each repository deploys from, from a table: magenta's <code>production</code> branch exists, but it has been dead since Nov 2025. | ||
** | ** The tool reads the branch's head itself, from GitHub (until the flip) or githouse, never from memory-lane, and shows <code>git log</code> and the diffstat of what will be merged. A signature covers only what the signer was shown; if memory-lane supplied the head, a compromised memory-lane could show one diff and hand over another hash (re). | ||
** | ** It signs the order with <code>ssh-keygen -Y sign</code> in a namespace of its own, <code>magenta-merge</code>, so no sign-in signature (namespace <code>magenta-moods</code>) can be replayed as a merge. The order carries a nonce and an expiry about 300 blocks out. | ||
* | ** It hands the order and signature to githouse through a mailbox at memory-lane (changed 8 Nov). Only Justin has ssh to maybelle, as root; skyler doesn't, and neither does a phone. So memory-lane keeps signed orders without reading them, and githouse fetches them every few seconds over the private network, checks each signature itself, and writes the outcome beside the repositories as before. The signature makes the carrier irrelevant (re, 7 Nov). A compromised memory-lane can delay or drop an order, never forge one, and the nonce stops it from replaying one. | ||
* | * '''githouse checks''', against a signers file of its own for the <code>magenta-merge</code> namespace, which ansible writes from the inventory and only the people who may merge are in (Justin's call; to start: justin). It refuses an expired order, a nonce already used (each is kept), a branch whose head has moved, a target that isn't one of GitHub's own branches, and a conflict. The signer chooses the target, and signs it with the rest; <code>magenta.sh merge</code> defaults it to the work's deploy branch (re's review of maybelle-config#286). | ||
* | * '''The merge''' is <code>git merge --no-ff</code> of that commit into its target (<code>production</code> where there is one, else <code>main</code>), with the message "Merge <repo>:<branch> (<commit>)" and who signed. '''Until the flip it is pushed to GitHub first''', with the deploy key: githouse force-fetches the deploy branches from GitHub every minute, so a merge written into githouse first is overwritten (re). The fetch brings it back; the name moves to <code>refs/merged/</code>. GitHub marks a pull request merged when its head reaches the base, so an open one closes itself. The signed order and its outcome are written beside the repositories, and memory-lane, which mounts them read-only, tells the Mood. | ||
* | * '''From a phone, not yet.''' A passkey on magenta's own page is not SSH strength: Face ID shows the site, not the order, and the page's script, served by memory-lane, sets what is signed. Enrolled by an SSH signature, the key enrolled is one nobody can check by eye and memory-lane carries it, so a compromised memory-lane could enroll its own and merge whenever it liked (re). The honest options: githouse serves a small page of its own for passkeys, which puts githouse on the web (section 3 rules that out today); or the plan says plainly that a phone merge trusts memory-lane at the moment of the press. Justin's choice; the terminal path comes first either way. '''Chosen 7 Nov: the second.''' A phone merge trusts memory-lane at the moment of the press, and the brakes of section 10 are what make that acceptable. | ||
* '''Not yet:''' requiring Jenkins to be green first. | |||
== 9. A place to discuss each piece of work (proposed | == 9. A place to discuss each piece of work (proposed 7 Nov; built 9 Nov) == | ||
'''Built:''' each candidate's page, at <code>/ramp/<repo>/<branch>/</code>: a Tiding with its description, commits, each file's diff, the whole file with every line addressable, the 👍s and Merge ([https://github.com/cryptograss/magenta/pull/448 magenta#448]). What's said on it shows in its Mood, and replies there show on it ([https://github.com/cryptograss/magenta/pull/449 magenta#449]); the page moved into the ramp with [https://github.com/cryptograss/magenta/pull/452 magenta#452]. A branch without a pull request takes its title from its newest commit's subject and has no description of its own; its page shows every commit with its whole message. The home for such a description, above, is still open. | |||
Justin, 7 Nov: each pull request, and each mergeable, gets a page to talk about it, built on Tidings' interface, which needs upgrading anyway. On it: the description, the commits, the 👍s and "merge now" (section 8), and the conversation. The description of a branch with no pull request needs a home; one candidate is a card of its own on PickiPedia, written by its author. To be designed after reading how Tidings keeps its comments. | |||
== 10. What a merge can do, and when a key is stolen (decided in part, 7 Nov) == | == 10. What a merge can do, and when a key is stolen (decided in part, 7 Nov) == | ||
Justin's answers, 7 Nov ([https://magenta.cryptograss.live/moods/ramp/#m-39e5f25f-0450-463c-931a-9134400badeb #m-39e5f25f]): merging from the phone, for every repository, pickipedia, arthel and maybelle-config included. The mergers are justin and skyler. No second admin for now. His one concern: whoever holds a merger's key or phone session must never be able to stop him from kicking them off. Today he can do that by removing them from the GitHub org. | |||
'''What a merge does, by repository''' (checked 7 Nov in maybelle-config <code>maybelle/jobs/</code>): | |||
* | * '''pickipedia, arthel: a merge runs code.''' <code>pickipedia-build</code> and <code>arthel-production-build</code> check out <code>production</code> every 5 and 2 minutes and run the Jenkinsfile in the merged commit, on maybelle's Jenkins, with its credentials. pickipedia's build is staged and then deployed to the wiki. | ||
* | * '''maybelle-config: a merge is deploy logic.''' It runs as root on every server at the next redeploy, which a person starts. | ||
* | * '''magenta, rabbithole, pickipedia-mcp: a merge does nothing on its own.''' Nothing runs until a person redeploys. | ||
'''One tier.''' Any merger may merge any repository, from a signed-in device or with <code>magenta.sh merge</code>. For a press on a device, memory-lane signs the order with a merge key of its own (principal <code>memory-lane</code>), naming whose press it was. githouse records both. A compromised memory-lane can then merge anything, including pickipedia and arthel, which run within minutes. Justin accepted that trade (7 Nov) for merging from the phone. | |||
'''The rule that stays: no merge can take away Justin's way to stop it.''' | |||
* | * '''Root is the anchor.''' No role in maybelle-config writes root's <code>authorized_keys</code> on any server (checked 7 Nov; the roles that write keys write only the recovery, recovery-log, ssh-router and container users'). So Justin's root SSH and Hetzner's console sit outside anything a merge can change. They play the part that owning the GitHub org plays today. | ||
* {{Bot_proposes|'''The ban reaches githouse before the phone button exists.''' Today, removing someone from the org stops their merging. githouse pushes to GitHub with its own deploy key, so once merges go through githouse, the org no longer stops anyone. So githouse asks memory-lane's <code>/api/brakes/</code> at every order whether AZ5 is on and whether the signer is banned, and refuses unless both answers are an explicit no. Device orders will name whoever pressed, and be asked about too. Only an admin (MOOD_ADMINS: justin) bans or unbans. If memory-lane doesn't answer, githouse merges nothing until it does. A braked order's nonce is spent, so it can't be replayed once the brake is off. ''Built 8 Nov: maybelle-config#286 (<code>61e88fe8</code>) and magenta's <code>/api/brakes/</code>.'' | * {{Bot_proposes|'''The ban reaches githouse before the phone button exists.''' Today, removing someone from the org stops their merging. githouse pushes to GitHub with its own deploy key, so once merges go through githouse, the org no longer stops anyone. So githouse asks memory-lane's <code>/api/brakes/</code> at every order whether AZ5 is on and whether the signer is banned, and refuses unless both answers are an explicit no. Device orders will name whoever pressed, and be asked about too. Only an admin (MOOD_ADMINS: justin) bans or unbans. If memory-lane doesn't answer, githouse merges nothing until it does. A braked order's nonce is spent, so it can't be replayed once the brake is off. ''Built 8 Nov: maybelle-config#286 (<code>61e88fe8</code>) and magenta's <code>/api/brakes/</code>.'' | ||
* '''A freeze that trusts nothing.''' memory-lane holds a merge key, so a compromised memory-lane could also hide a ban. <code>/mnt/persist/githouse/etc/frozen</code> on maybelle stops every merge, whatever memory-lane says. githouse sees that directory read-only, so only root on maybelle makes or removes the file, and a redeploy leaves it alone. ''Built 8 Nov, same PR; <code>magenta.sh freeze</code> is planned as a wrapper.'' | * '''A freeze that trusts nothing.''' memory-lane holds a merge key, so a compromised memory-lane could also hide a ban. <code>/mnt/persist/githouse/etc/frozen</code> on maybelle stops every merge, whatever memory-lane says. githouse sees that directory read-only, so only root on maybelle makes or removes the file, and a redeploy leaves it alone. ''Built 8 Nov, same PR; <code>magenta.sh freeze</code> is planned as a wrapper.'' | ||
* | * '''A ban reaches hunter's router too''', so a stolen key can't go on steering re and blu with full tools (both list skyler in <code>people</code>). | ||
* | * '''A redeploy shows a change in who holds access.''' A merge to maybelle-config can rewrite the people, their keys, the mergers or MOOD_ADMINS, and Justin's next routine redeploy would apply it without his noticing. Before it runs, a redeploy prints any change in those since what is running, using <code>scripts/render_diff.py</code>. | ||
* | * '''Open: Jenkins' GitHub token.''' pickipedia and arthel merges run in Jenkins, which holds <code>maybelle_github_token</code>. If that token can administer the org, a merge there can change the org itself, and with it the deploy keys. Which account it belongs to and what scopes it has is for a person to check, without printing it. | ||
'''Build order.''' 1. The ban and the freeze in githouse. 2. <code>magenta.sh merge</code>, with skyler in <code>githouse_mergers</code>. 3. The phone button. Bans at the router and the access diff at redeploy can go alongside any of these. Nothing gets easier to merge until 1 is live. | |||
A runbook, in order, at [[Cryptograss:Runbooks/A key is compromised]]. | |||
'''As of 10 Nov:''' build order 1, 2 and 3 are done (the brakes, 8 Nov; <code>magenta.sh merge</code> with skyler a signer, 8 Nov; the phone button, 9 Nov). Not built: the ban at hunter's router, the access diff at redeploy, and <code>magenta.sh freeze</code>. Not checked: Jenkins' GitHub token. The runbook hasn't been verified. | |||
== 11. Branch tests on Jenkins, without its secrets (proposed, 9 Nov; revised after re's review) == | == 11. Branch tests on Jenkins, without its secrets (proposed, 9 Nov; revised after re's review) == | ||
Justin, 9 Nov (#ramp, [https://magenta.cryptograss.live/moods/ramp/#m-26738398-6ab5-4aae-84f2-f478af2e2c1d 00:42Z] and [https://magenta.cryptograss.live/moods/ramp/#m-3af6ccb9-d97e-4ef0-a9fc-2aee056409c7 00:46Z]): run the tests on every repository's branches, not only arthel's; keep Jenkins, because its console is the point, especially for a failed run; and Jenkins is public by design, so a sneaky branch must not be able to print secrets into it. | |||
'''Where things stand''' (maybelle-config <code>production</code>, checked 9 Nov, names only): every build runs on Jenkins' built-in node (<code>numExecutors: 3</code>, no agents), as SYSTEM (no Authorize Project). The container's environment holds <code>JENKINS_ADMIN_PASSWORD</code>, <code>JENKINS_REPORTER_PASSWORD</code>, <code>GITHUB_TOKEN</code>, the Blue Railroad bot's login, <code>DELIVERY_KID_API_KEY</code> and <code>MOOD_ALARM_KEY</code>, and the whole <code>maybelle_env</code> vault blob through <code>env_file</code> (<code>maybelle.yml</code> 122, 324; re). <code>anonymous</code> has <code>Job/Read</code>, so every console is public. Today only merged code (pickipedia and arthel <code>production</code>) and arthel's <code>build-on-maybelle</code> branch reach a build. | |||
'''Why emptying the environment isn't enough.''' A build on the built-in node runs as the controller's own user, on its own disk: it can read <code>jenkins_home</code>, where every credential sits beside the key that decrypts it. And a branch's Jenkinsfile chooses its own node (<code>agent { label 'built-in' }</code>), whatever the node's mode; its Groovy runs on the controller, in a sandbox with a long history of escapes (re). So the line is: '''a branch's code never chooses where it runs, and never runs on the controller.''' | |||
'''The design:''' | |||
# '''One fixed pipeline per repository, written in maybelle-config, never the branch's Jenkinsfile''' (re). githouse sees every push, so it triggers the repository's job with the repository, branch and commit. The fixed script clones that commit from GitHub anonymously, on the <code>branches</code> agent, and runs the repository's test command. Branch code only ever runs inside <code>sh</code> on the agent; the console stays public. It also needs no multibranch scan, which anonymously would hit GitHub's 60 an hour at once. | |||
# '''An agent for branches, label <code>branches</code>, with no secrets.''' Where, Justin's call: | |||
#* '''A separate small box''' (re's preference, and mine): Hetzner's firewall keeps it off maybelle's private network, and the agent dials Jenkins over its public websocket. On maybelle, untrusted code would share a kernel with githouse's deploy key and memory-lane's merge key: one container escape reaches both. | |||
#* '''On maybelle''': only with an <code>internal: true</code> Docker network and an egress proxy allowing npm, PyPI and GitHub, or iptables rules on both FORWARD and INPUT. By default a container reaches every address of its host (INPUT, which <code>DOCKER-USER</code> never sees: memory-lane, githouse and postgres on 10.0.0.2), the rest of the private network (pickipedia, 10.0.0.4) and Hetzner's metadata service (169.254.169.254). | |||
#* Either way, the deploy runs a check inside the agent that fails if any of those addresses answers. | |||
# '''Credentials scoped to the trusted jobs.''' They move from the global store into a folder holding only <code>pickipedia-build</code>, <code>arthel-production-build</code> and the like; <code>withCredentials</code> finds only credentials in a job's own folder and those above it. | |||
# '''Authorize Project''': branch jobs run as anonymous, so they can't <code>build job:</code> a trusted one. Today that would be harmless only by luck (the trusted jobs take no parameters, <code>deploy-hunter</code> only reporting ones); trusted jobs stay free of parameters. | |||
# '''The controller's environment emptied, all of it''' (the list above and the <code>env_file</code>): secrets reach Configuration as Code through files only it reads, so no trusted build prints one by mistake. | |||
'''What stays trusted:''' merged code. A merge into pickipedia or arthel <code>production</code> still runs on the built-in node with its credentials; that's what the brakes and the mergers list (section 10) are for. | |||
re's lane (section 7): reviewed by re, 9 Nov 00:49Z; folded in above. | |||
'''Built, 9 Nov:''' the <code>branches</code> agent on maybelle (Justin's choice, 01:00Z), behind an internal network and egress proxy, its isolation checked at every deploy; one fixed pipeline per repository, started by githouse for every push, running <code>ci/test</code> as an unprivileged user ([https://github.com/cryptograss/maybelle-config/pull/292 maybelle-config#292] to #295, [https://github.com/cryptograss/magenta/pull/443 magenta#443]); the reporter may start only <code>branch-tests</code> and <code>deploy-hunter</code> (#296); the ramp shows each branch's red or green. 10 Nov: the agent has an init as PID 1, and an aborted build no longer jams the next (<code>magent/branches-agent-init</code>). Dropped: moving credentials into a folder (Justin, 9 Nov 10:58Z). Moved to ''#security'': emptying the controller's environment, and the trusted agent it needs to mean much. | |||
Latest revision as of 23:13, 10 November 2026
The ramp: how work gets from a branch to a running server, with a record of each step, and less of it depending on GitHub. Settled in Cryptograss:Moods/ramp between 30 October and 5 November 2026 (Justin's voice memo of 30 Oct, then re and magent). What's open is on Cryptograss:Moods/ramp/todo. Dates are as said; blocks where known.
1. A mergeable is a branch, named once
A piece of work is <repo>:<branch>, as maybelle-config:re/fix-mtime-guard, for its whole life. Not a pull request number: a number names nothing until GitHub gives it, and one piece of work across two repositories is still one thing (pickipedia#168 and pickipedia-mcp#43 were one change). A commit hash says which version of the work was looked at; an ask or a watch records both.
- Branches are prefixed with their owner:
re/…,magent/…,justin/…. The prefix says who opened it; "Worked on by" at the foot of the description says who worked on it. - A name is used once. When the branch merges, its ref moves to
refs/merged/<branch>; when its owner deletes it, torefs/closed/<branch>. A push that makes either name again is refused and offered the next free one (re/foo-2). The bare repository is its own registry. - "Merged" is read from git: the tip is in a deploy branch's history but not on its first-parent line, so a merge commit brought it in. This needs merge commits: squash and rebase merges, and GitHub's merge queue, break it (settled 31 Oct; the six repositories githouse copies were set to merge commits only by 7 Nov 00:39Z, section 5).
- Autocomplete in the Mood composer, from the start (Justin, 1 Nov): magenta#333, typing
repo:orre/.
2. Records on PickiPedia, the view on magenta
- Each pull request has a card at
VersionControl:PRs/<repo>/<number>(theVersionControl:namespace, aliasVC:, pickipedia#168; exempt from the verification gate, pickipedia-mcp#43: a card is GitHub's word mirrored, not a claim). Template:PR sets the properties; magenta writes the cards (magenta#314, live since 31 Oct). The foot of a description, as Cryptograss:Emergents/GeneralOrders has it, givesDeploy,Worked on byand the Moods. Proof:answers. - The view is a page on magenta, because "the last two messages mentioning it" is magenta's data: every open pull request across our repositories (and a few followed from outside), the last five merged, who worked on it and where, sorted by last push or last mention, with a temperature from last activity, size and review state (re, 30 Oct). Built: the ramp, at
/ramp/(magenta#420, 7 Nov, then #421 and the 9-10 Nov branches): every open candidate branch, with or without a pull request; its tests' red or green; pushed in blocks ago; the column heads sort it; "Merged lately" lists every branch githouse merged. A candidate named in a message (magenta:re/foo,magenta#7) links to its page there. - When cards name branches instead of numbers (section 1), the number becomes a property of the card, like an ISBN.
3. githouse: our canonical copies, on maybelle
Justin, 2 Nov: the canonical copy of each repository, the one deploys come from, lives on maybelle, not hunter. Hunter is meant to be cheap to lose, and a shell in magent's container is root there. Design: magent 2 Nov, re's ten points the same day (#ramp, 16:24Z). Built: maybelle-config#250, reworked to put githouse on maybelle and its mirror on hunter, merged and deployed 6 Nov 05:01Z (then #280, the locked git account).
- On maybelle: one bare repository per repo. GitHub's branches (
main,master,production,release/*) and pull requests' heads are fetched every 60 seconds, repositories spread across the minute, nothing fetched when nothing changed. Until the flip, those refs are forced and pruned to match GitHub; nobody pushes them here. After it, githouse owns the deploy branches and only compares GitHub's (section 6); pull requests' heads stay GitHub's. Until the flip (section 6), a merge reaches GitHub first: githouse's own merges (section 8) are pushed there and fetched back, and maybelle's copy lags GitHub by up to a minute. - Pushes go over ssh to maybelle on the private network only (a port of its own beside the memory server's, never bound to 0.0.0.0: the role asserts it). Each key's forced command names who it is; one may push only
<own name>/*, no tags, never a deploy branch. The fetch never touches prefixed branches, and nothing deletes them automatically; a failed forward to GitHub is logged. Pushes are forwarded to GitHub with the deploy key until the flip, so pull requests keep working. - A fix to another's branch is a branch of your own, merged in by the owner (re and magent, 7 Nov: githouse refused re's push to
magent/asks-from-githouse, as designed; herre/fix-asks-and-githousewas merged into it with a merge commit, so the record shows whose commit was whose). Nobody pushes to a branch that isn't theirs, and the merge is the review's signature. - Keys are made in each emergent's container (Justin, 1 Nov: not a vault secret; it means nothing outside hunter). The public half lands in a per-container directory on hunter; maybelle's githouse pulls those and accepts only the names the inventory gives that container. Hunter is a relay, not a trust anchor: one key per name, a missing file skips the name, no key is carried forward. Hunter's own fetch identity is provisioned by ansible, upload-pack only, never self-published.
- Hunter keeps a read-only copy, fetched from maybelle every minute and mounted at
/srv/gitin every container, so clones are instant (--reference-if-able). Nothing on it needs backing up. - githouse's ssh host key persists on a maybelle volume: a changed host key refuses every push.
branches.json(every repository's branches: live, merged, closed) is served by memory-lane at/api/branches/, read from its read-only mount of githouse (magenta#333): maybelle's Caddy publishes nothing of githouse itself.
4. Watches: merged is a commit, deployed is a commit
Justin, 31 Oct: a standing list of watches, each (repository, commit, deploy branch) and the ask it came from, kept in magenta's database. A watch resolves when the commit is an ancestor of the deploy branch after a fetch: an unambiguous confirmation of the merge of an exact commit, with GitHub's PR taxonomy and webhook as optional extras. If the branch got more commits before merging, the resolution says so. People and emergents retarget, wontfix or clear a watch, recorded with who and when, as asks are.
- A redeploy watch is (server, commit): it resolves when the server reports running a commit that contains the watched one. Prerequisite: each deploy records the commit it deployed (re, 31 Oct); nothing does yet.
- Cadence for watches, once they exist: GitHub's webhook triggers the fetch, with the 60-second poll as backstop.
- Built, the merge half (as of 10 Nov): asks and to-do items are the watches, not a table of their own. A merge ask resolves when githouse's copy has its pull request's head in a deploy branch, or its branch's name in
refs/merged/(magenta#415); a to-do item linking a candidate's page,/ramp/<repo>/<branch>/, ticks itself when githouse merges or closes the branch, and the wiki bot ticks it on the page (magent/todo-branch-merged, 10 Nov). A branch closed unmerged answers its ask. The Mood an order came from gets a "✓ merged" line (magenta#444). - Next: the deploy half. Each deploy stamps the commit it deployed; the ramp shows "live on <server>", and a redeploy ask resolves on that commit, not on any deploy finishing (10 Nov: "is maybelle live?" was answered by reading the served page, once wrongly).
5. Repository settings (GitHub, Justin's)
Settled 6 Nov (Justin, 11:22Z and 19:57Z): as little of GitHub as the ramp needs, and nothing that entrenches it.
mainstays the default branch everywhere: the branch where history is final. The branch that deploys is per-repository configuration (productionfor maybelle-config, pickipedia and arthel;mainfor magenta), not GitHub's HEAD; githouse fetches deploy branches by name. The mains catch up to the productions after the ramp settles: arthel and maybelle-config are fast-forwards, pickipedia's main holds 2 commits production lacks, magenta'sproductionis dead (a634f16, Nov 2025). Measured on the mirror, 6 Nov.- Merge commits only on the six repositories githouse copies: squash and rebase off. This is the one setting the ramp needs, because githouse reads "merged" from git (section 1) rather than from GitHub's API; a squash or rebase leaves no trace of the pushed tip, so the name retires as closed and a watch never resolves. Our practice already: every pull request merged into the four deploy branches in the last 30 first-parent commits was a merge commit. Moot after the flip, when githouse merges with
--no-ff. Merge queue is a ruleset feature; with no rulesets there is nothing to turn off. - No rulesets. The deploy branches are guarded by githouse's hook, not by GitHub: a push to
productionor to another's prefix is refused before it lands (proved from magent's container, 6 Nov 11:26Z). The only paths straight to a deploy branch on GitHub are Justin's account and the deploy key, and at the flip GitHub becomes a mirror we push to, which a ruleset would have to be undone for.
6. The flip, and backups
When githouse is the place merges happen (a ramp merge ask, carried out by a person's tool on maybelle), GitHub becomes a mirror we push to, and issues move to PickiPedia after pull request cards have proved out (re's phases, 30 Oct). Not before.
- What only githouse holds and a backup must carry (re and magent, 1 Nov):
refs/merged/andrefs/closed/(the name registry), the ssh host key, and any pushed branch whose forward to GitHub failed. GitHub's branches come back from GitHub. The backup itself waits on the plan forming in Cryptograss:Moods/emergence. - As of 10 Nov: nothing in maybelle-config backs up
/mnt/persist/githouse. Hunter's mirror fetchesrefs/merged/andrefs/closed/every minute, so the name registry has a second copy there; the host key, and a branch whose forward failed, have none. - The flip: merged and live, 10 Nov (maybelle-config
81ebc2c6, maybelle deployed 22:10Z;magent/githouse-owns,re/githouse-owns-new-names,magent/jenkins-from-githouseandmagent/githouse-secret-check, withmagent/jenkins-gitconfig-fileafter the first deploy failed). Checked at block 26,387,713: hunter's mirror holdsrefs/github/productionbesideproductionin maybelle-config, arthel and pickipedia, which only the new githouse writes, and each pair matches.[unverified]- githouse owns
main,master,productionandrelease/*. They move only by a signed merge there, then are pushed to GitHub, never forced. - GitHub's copy, moved around githouse (a push, or GitHub's merge button), is never taken. Behind ours, githouse sends ours again; ahead or apart, it's logged and written to
github.json. A deploy-branch name new on GitHub is logged too, not taken: only a brand-new githouse takes GitHub's, once. Root on maybelle takes a fast-forward withdocker exec -u git githouse githouse adopt <repo> <branch>. - So GitHub's merge button deploys nothing. Merge from the pill, or with
magenta.sh merge. - Deploys come from githouse: maybelle's magenta, the deploy scripts (hunter, pickipedia, delivery-kid, chapter 1), hunter's build image, and Jenkins' pickipedia-build, arthel-production-build, arthel-fetch-chain-data and arthel-fetch-pinata-data (section 7). GitHub only on a cold start (githouse has no copy yet), and the deploy says so; a failed fetch stops the deploy.
- A push carrying a secret is refused at githouse's door: a vault value (githouse holds only their SHA-256 hashes) or a known secret's shape, in any added line, message or file name, merges included; octopus merges are refused.
githouse scanlooks through all of history.[unverified]
- githouse owns
- justinholmes.com's shows (happenings): since
arthel:magent/happenings-pinnedmerged (10 Nov), arthel builds the happenings commit itsproductionpins, not happenings' newest. A show change is a commit to happenings, then an arthel merge that moves the pin. That merge shows only a hash: read happenings' log between the old pin and the new one before signing. happenings stays out of githouse because it's private, and githouse's copies are readable from every container on hunter (re, magent, Skyler, #ramp 18:01-19:12Z).[unverified] - GitHub stays, as a mirror: the cold-start bootstrap and the off-site copy of the deploy branches, the public face, dependabot, and the pull request cards. Retiring it would need a replacement for each.
- Where this ends: nothing reaches a server unless a merge signed in githouse put it there. What's left, as of 10 Nov 23:1xZ (each is on Cryptograss:Moods/ramp/todo):[unverified]
- still deployed from GitHub: rabbithole (pickipedia VPS), pickipedia-mcp (hunter's image), blue-railroad-import (Jenkins' image; not in githouse);
arthel-branches-builds: it built any arthel branch on GitHub with that branch's own Jenkinsfile, on the controller, emergents' branches included (it builtmagent/happenings-pinned, 10 Nov), not onlybuild-on-maybelleas this page said before. Retired onmaybelle-config:magent/retire-arthel-branches-builds, not yet merged;[unverified]- Jenkins' GitHub token: unused once that's live; then deleted on GitHub and taken out of the vault;[unverified]
- the ramp shows
github.json, so a move around githouse is seen, not only logged; - the backup of
/mnt/persist/githouse, now the only primary copy of the deploy branches.
7. Jenkins (re's lane)
Where the tests run and what they are: re's map of the Jenkinsfiles (#ramp, 30 Oct 22:57Z). Done: the maybelle mtime guard (maybelle-config#244); Jenkins reds post to #ops and wake magent when they stay red (magenta#373, #402; maybelle-config#275, 5 Nov). Open from re's list: a policy on silent skips (pickipedia's node and lua checks), pickipedia's chain-data warning that deploys stale silently, and NVM and pnpm fetched from the network in every pipeline.
- Where the trusted jobs get their code, from the flip (section 6): pickipedia-build and the three arthel production jobs check out
file:///srv/git/<repo>.git, githouse's copies mounted read-only, with no GitHub token. Two settings make it work:hudson.plugins.git.GitSCM.ALLOW_LOCAL_CHECKOUTinJAVA_OPTS, and asafe.directoryper repository in jenkins's own git config. Allowing local checkout opens nothing new (re, 10 Nov): only admin configures jobs, and the one job that runs a Jenkinsfile from GitHub, arthel-branches-builds, already runsshon the controller. Still from GitHub: branch-tests, by design (its agent reaches only GitHub and the registries; githouse pushes before it asks for a test), arthel-branches-builds, and the happenings submodule, fetched at the commit arthel pins.
8. Merging a branch, without a pull request (proposed 7 Nov; built 8-10 Nov)
Built: magenta.sh merge and the mailbox (magenta#428, maybelle-config#287, 8 Nov); the Merge button on every candidate's sheet, its order signed by memory-lane for whoever pressed (magenta#433, maybelle-config#289, 9 Nov; first press 00:38Z); the bar under the nav that follows a press to ✓ or ✗ on every page (10 Nov). The signers are justin and skyler (githouse_mergers, 53a17642). Still not: requiring the branch's tests to be green first.
Justin, 7 Nov: build it, with every merge approved by a person at the SSH tier, and two kinds of approval: a 👍 ("I like this") and "merge now" (what GitHub's merge button is today). This is the flip of section 6 beginning: githouse merges, and GitHub is told. Revised the same day after re's review (#ramp, 17:27Z).
- What is approved is a commit, not a name. An order names
<repo>:<branch>, its head commit and the branch it goes into. "Merge now" merges exactly that commit, or nothing. - 👍 comes from any person's signed-in device: one press, recorded with who, when and which commit, and shown on the ramp (stale once the branch moves). It is a signal only. githouse never requires one, because a 👍 is only memory-lane's say-so.
- Merge now, from a terminal:
magenta.sh merge pickipedia:re/foo. memory-lane carries the order and decides nothing in it. Built 8 Nov: magenta#428, maybelle-config#287. The target defaults to the branch each repository deploys from, from a table: magenta'sproductionbranch exists, but it has been dead since Nov 2025.- The tool reads the branch's head itself, from GitHub (until the flip) or githouse, never from memory-lane, and shows
git logand the diffstat of what will be merged. A signature covers only what the signer was shown; if memory-lane supplied the head, a compromised memory-lane could show one diff and hand over another hash (re). - It signs the order with
ssh-keygen -Y signin a namespace of its own,magenta-merge, so no sign-in signature (namespacemagenta-moods) can be replayed as a merge. The order carries a nonce and an expiry about 300 blocks out. - It hands the order and signature to githouse through a mailbox at memory-lane (changed 8 Nov). Only Justin has ssh to maybelle, as root; skyler doesn't, and neither does a phone. So memory-lane keeps signed orders without reading them, and githouse fetches them every few seconds over the private network, checks each signature itself, and writes the outcome beside the repositories as before. The signature makes the carrier irrelevant (re, 7 Nov). A compromised memory-lane can delay or drop an order, never forge one, and the nonce stops it from replaying one.
- The tool reads the branch's head itself, from GitHub (until the flip) or githouse, never from memory-lane, and shows
- githouse checks, against a signers file of its own for the
magenta-mergenamespace, which ansible writes from the inventory and only the people who may merge are in (Justin's call; to start: justin). It refuses an expired order, a nonce already used (each is kept), a branch whose head has moved, a target that isn't one of GitHub's own branches, and a conflict. The signer chooses the target, and signs it with the rest;magenta.sh mergedefaults it to the work's deploy branch (re's review of maybelle-config#286). - The merge is
git merge --no-ffof that commit into its target (productionwhere there is one, elsemain), with the message "Merge <repo>:<branch> (<commit>)" and who signed. Until the flip it is pushed to GitHub first, with the deploy key: githouse force-fetches the deploy branches from GitHub every minute, so a merge written into githouse first is overwritten (re). The fetch brings it back; the name moves torefs/merged/. GitHub marks a pull request merged when its head reaches the base, so an open one closes itself. The signed order and its outcome are written beside the repositories, and memory-lane, which mounts them read-only, tells the Mood. - From a phone, not yet. A passkey on magenta's own page is not SSH strength: Face ID shows the site, not the order, and the page's script, served by memory-lane, sets what is signed. Enrolled by an SSH signature, the key enrolled is one nobody can check by eye and memory-lane carries it, so a compromised memory-lane could enroll its own and merge whenever it liked (re). The honest options: githouse serves a small page of its own for passkeys, which puts githouse on the web (section 3 rules that out today); or the plan says plainly that a phone merge trusts memory-lane at the moment of the press. Justin's choice; the terminal path comes first either way. Chosen 7 Nov: the second. A phone merge trusts memory-lane at the moment of the press, and the brakes of section 10 are what make that acceptable.
- Not yet: requiring Jenkins to be green first.
9. A place to discuss each piece of work (proposed 7 Nov; built 9 Nov)
Built: each candidate's page, at /ramp/<repo>/<branch>/: a Tiding with its description, commits, each file's diff, the whole file with every line addressable, the 👍s and Merge (magenta#448). What's said on it shows in its Mood, and replies there show on it (magenta#449); the page moved into the ramp with magenta#452. A branch without a pull request takes its title from its newest commit's subject and has no description of its own; its page shows every commit with its whole message. The home for such a description, above, is still open.
Justin, 7 Nov: each pull request, and each mergeable, gets a page to talk about it, built on Tidings' interface, which needs upgrading anyway. On it: the description, the commits, the 👍s and "merge now" (section 8), and the conversation. The description of a branch with no pull request needs a home; one candidate is a card of its own on PickiPedia, written by its author. To be designed after reading how Tidings keeps its comments.
10. What a merge can do, and when a key is stolen (decided in part, 7 Nov)
Justin's answers, 7 Nov (#m-39e5f25f): merging from the phone, for every repository, pickipedia, arthel and maybelle-config included. The mergers are justin and skyler. No second admin for now. His one concern: whoever holds a merger's key or phone session must never be able to stop him from kicking them off. Today he can do that by removing them from the GitHub org.
What a merge does, by repository (checked 7 Nov in maybelle-config maybelle/jobs/):
- pickipedia, arthel: a merge runs code.
pickipedia-buildandarthel-production-buildcheck outproductionevery 5 and 2 minutes and run the Jenkinsfile in the merged commit, on maybelle's Jenkins, with its credentials. pickipedia's build is staged and then deployed to the wiki. - maybelle-config: a merge is deploy logic. It runs as root on every server at the next redeploy, which a person starts.
- magenta, rabbithole, pickipedia-mcp: a merge does nothing on its own. Nothing runs until a person redeploys.
One tier. Any merger may merge any repository, from a signed-in device or with magenta.sh merge. For a press on a device, memory-lane signs the order with a merge key of its own (principal memory-lane), naming whose press it was. githouse records both. A compromised memory-lane can then merge anything, including pickipedia and arthel, which run within minutes. Justin accepted that trade (7 Nov) for merging from the phone.
The rule that stays: no merge can take away Justin's way to stop it.
- Root is the anchor. No role in maybelle-config writes root's
authorized_keyson any server (checked 7 Nov; the roles that write keys write only the recovery, recovery-log, ssh-router and container users'). So Justin's root SSH and Hetzner's console sit outside anything a merge can change. They play the part that owning the GitHub org plays today. - {{Bot_proposes|The ban reaches githouse before the phone button exists. Today, removing someone from the org stops their merging. githouse pushes to GitHub with its own deploy key, so once merges go through githouse, the org no longer stops anyone. So githouse asks memory-lane's
/api/brakes/at every order whether AZ5 is on and whether the signer is banned, and refuses unless both answers are an explicit no. Device orders will name whoever pressed, and be asked about too. Only an admin (MOOD_ADMINS: justin) bans or unbans. If memory-lane doesn't answer, githouse merges nothing until it does. A braked order's nonce is spent, so it can't be replayed once the brake is off. Built 8 Nov: maybelle-config#286 (61e88fe8) and magenta's/api/brakes/. - A freeze that trusts nothing. memory-lane holds a merge key, so a compromised memory-lane could also hide a ban.
/mnt/persist/githouse/etc/frozenon maybelle stops every merge, whatever memory-lane says. githouse sees that directory read-only, so only root on maybelle makes or removes the file, and a redeploy leaves it alone. Built 8 Nov, same PR;magenta.sh freezeis planned as a wrapper. - A ban reaches hunter's router too, so a stolen key can't go on steering re and blu with full tools (both list skyler in
people). - A redeploy shows a change in who holds access. A merge to maybelle-config can rewrite the people, their keys, the mergers or MOOD_ADMINS, and Justin's next routine redeploy would apply it without his noticing. Before it runs, a redeploy prints any change in those since what is running, using
scripts/render_diff.py. - Open: Jenkins' GitHub token. pickipedia and arthel merges run in Jenkins, which holds
maybelle_github_token. If that token can administer the org, a merge there can change the org itself, and with it the deploy keys. Which account it belongs to and what scopes it has is for a person to check, without printing it.
Build order. 1. The ban and the freeze in githouse. 2. magenta.sh merge, with skyler in githouse_mergers. 3. The phone button. Bans at the router and the access diff at redeploy can go alongside any of these. Nothing gets easier to merge until 1 is live.
A runbook, in order, at Cryptograss:Runbooks/A key is compromised.
As of 10 Nov: build order 1, 2 and 3 are done (the brakes, 8 Nov; magenta.sh merge with skyler a signer, 8 Nov; the phone button, 9 Nov). Not built: the ban at hunter's router, the access diff at redeploy, and magenta.sh freeze. Not checked: Jenkins' GitHub token. The runbook hasn't been verified.
11. Branch tests on Jenkins, without its secrets (proposed, 9 Nov; revised after re's review)
Justin, 9 Nov (#ramp, 00:42Z and 00:46Z): run the tests on every repository's branches, not only arthel's; keep Jenkins, because its console is the point, especially for a failed run; and Jenkins is public by design, so a sneaky branch must not be able to print secrets into it.
Where things stand (maybelle-config production, checked 9 Nov, names only): every build runs on Jenkins' built-in node (numExecutors: 3, no agents), as SYSTEM (no Authorize Project). The container's environment holds JENKINS_ADMIN_PASSWORD, JENKINS_REPORTER_PASSWORD, GITHUB_TOKEN, the Blue Railroad bot's login, DELIVERY_KID_API_KEY and MOOD_ALARM_KEY, and the whole maybelle_env vault blob through env_file (maybelle.yml 122, 324; re). anonymous has Job/Read, so every console is public. Today only merged code (pickipedia and arthel production) and arthel's build-on-maybelle branch reach a build.
Why emptying the environment isn't enough. A build on the built-in node runs as the controller's own user, on its own disk: it can read jenkins_home, where every credential sits beside the key that decrypts it. And a branch's Jenkinsfile chooses its own node (agent { label 'built-in' }), whatever the node's mode; its Groovy runs on the controller, in a sandbox with a long history of escapes (re). So the line is: a branch's code never chooses where it runs, and never runs on the controller.
The design:
- One fixed pipeline per repository, written in maybelle-config, never the branch's Jenkinsfile (re). githouse sees every push, so it triggers the repository's job with the repository, branch and commit. The fixed script clones that commit from GitHub anonymously, on the
branchesagent, and runs the repository's test command. Branch code only ever runs insideshon the agent; the console stays public. It also needs no multibranch scan, which anonymously would hit GitHub's 60 an hour at once. - An agent for branches, label
branches, with no secrets. Where, Justin's call:- A separate small box (re's preference, and mine): Hetzner's firewall keeps it off maybelle's private network, and the agent dials Jenkins over its public websocket. On maybelle, untrusted code would share a kernel with githouse's deploy key and memory-lane's merge key: one container escape reaches both.
- On maybelle: only with an
internal: trueDocker network and an egress proxy allowing npm, PyPI and GitHub, or iptables rules on both FORWARD and INPUT. By default a container reaches every address of its host (INPUT, whichDOCKER-USERnever sees: memory-lane, githouse and postgres on 10.0.0.2), the rest of the private network (pickipedia, 10.0.0.4) and Hetzner's metadata service (169.254.169.254). - Either way, the deploy runs a check inside the agent that fails if any of those addresses answers.
- Credentials scoped to the trusted jobs. They move from the global store into a folder holding only
pickipedia-build,arthel-production-buildand the like;withCredentialsfinds only credentials in a job's own folder and those above it. - Authorize Project: branch jobs run as anonymous, so they can't
build job:a trusted one. Today that would be harmless only by luck (the trusted jobs take no parameters,deploy-hunteronly reporting ones); trusted jobs stay free of parameters. - The controller's environment emptied, all of it (the list above and the
env_file): secrets reach Configuration as Code through files only it reads, so no trusted build prints one by mistake.
What stays trusted: merged code. A merge into pickipedia or arthel production still runs on the built-in node with its credentials; that's what the brakes and the mergers list (section 10) are for.
re's lane (section 7): reviewed by re, 9 Nov 00:49Z; folded in above.
Built, 9 Nov: the branches agent on maybelle (Justin's choice, 01:00Z), behind an internal network and egress proxy, its isolation checked at every deploy; one fixed pipeline per repository, started by githouse for every push, running ci/test as an unprivileged user (maybelle-config#292 to #295, magenta#443); the reporter may start only branch-tests and deploy-hunter (#296); the ramp shows each branch's red or green. 10 Nov: the agent has an init as PID 1, and an aborted build no longer jams the next (magent/branches-agent-init). Dropped: moving credentials into a folder (Justin, 9 Nov 10:58Z). Moved to #security: emptying the controller's environment, and the trusted agent it needs to mean much.