Cryptograss:Moods/ramp/todo

• ☐
  Redeploy hunter: magenta:re/candidates-news changes the poller

  deploy · justin

  On main since 73f095d (checked 10 Nov 04:48Z). Poller only, so hunter, not maybelle.
• ☐
  Redeploy delivery-kid: it runs 9fd62b2d (5 Nov), so the iPhone fix merged 9 Nov 17:39Z (strip-empty-sdtp) isn't live

  deploy · justin

  Checked /version 10 Nov 04:48Z. If maybelle-config:re/seeder-container (#290) and maybelle-config:magent/resdtp-releases (#298) merge first, one redeploy covers all three.
• ☐
  Verify Cryptograss:Moods/ramp/Plan: six revisions of magent's since your last verify (rev 14836, 6 Nov), in sections 8, 10 and 11

  verify · justin

  Section 8 (merging without a PR, as built), 10 (rev 15035: one tier, phone merges everywhere; no merge can take away your way to stop it), 11 (branch tests on Jenkins, after re's review). Newest is 15304, 9 Nov 00:50Z.
• ☐
  Verify Cryptograss:Runbooks/A key is compromised: never verified; all 8 revisions are magent's

  verify · justin

  Newest is 15094 (8 Nov 02:15Z: the brakes and AZ5 live). Its freeze brake is rev 15038.
• ☐
  Verify re's candidates-changed and candidates-same lines on Cryptograss:Emergents/Wake (rev 15582)

  verify · justin

  re's edit, 9 Nov 17:54Z, after your 17:42Z verify. Until verified, the poller sends the same words from poller/wake_texts.md.
• ☐
  Delete magenta's dead production branch on GitHub (a634f16, Nov 2025), or say it stays

  decision · justin

  Still there, checked 10 Nov 05:15Z. It's why the ramp measured every magenta branch with no pull request against a year of history (+85,583 lines). The ramp routes around it now; deleting it removes the trap for anything else that guesses production first.
• ☐
  Check which account maybelle_github_token (Jenkins' github-token) belongs to and its scopes, without printing it: pickipedia and arthel merges run code in Jenkins with it

  infra · justin

  If it can administer the org, a merge there can change the org and its deploy keys (Plan, section 10, open). Nothing in the record says it's been checked, as of 10 Nov.
• ☐
  hunter's disk is at 84% (warning at 85%): prune unused images in the hunter playbook

  infra · magent

  10 Nov 05:15Z: images 35.5 GB, 30.1 GB of it unused. The build cache is already capped (20 GB after a build, a weekly prune); images are never pruned. On 7 Nov 02:37Z it filled and the mirror's fetches failed until 03:42Z. A maybelle-config PR, not a hand prune.
• ☐
  hunter's ssh router refuses a banned person's key; a redeploy prints any change in people, keys, githouse_mergers or MOOD_ADMINS (scripts/render_diff.py)

  code · magent

  Plan section 10. Not built: neither is on maybelle-config production (52dc3a2c), checked 10 Nov.
• ☐
  A deploy watch (server, commit): deploys stamp the commit they deployed, and the Mood is told

  code · magent

  Plan, section 4. The merge half is done (#415; #444 says githouse's merges in their Mood). Tonight 'is maybelle live?' was answered by grepping the served page; a stamp makes it one look.
• ☐
  Deploys clone from githouse's copy on maybelle instead of GitHub (Jenkinsfiles, deploy scripts)

  code · magent

  re's point 9: its own PR, after githouse has run a few days.
• ☐
  Jenkins audit, re's open items: a policy on silent skips (pickipedia node/lua checks), pickipedia's chain-data warning that deploys stale silently, NVM/pnpm fetched from the network in every pipeline

  code · re

  From re's test map, 30 Oct. The mtime guard (#244) and the alarms (#402, #275) are done.

Done (64)

• ☑
  Merge magenta:magent/branch-links (a candidate named in a message, magenta:re/foo or magenta#7, links to its page in the ramp), then redeploy maybelle

  merge · justin

  Merged as e563759, 10 Nov ~05:23Z; live after maybelle's redeploy. It didn't tick itself: only GitHub pull request links did (fixed by the item below).
• ☑
  Merge magenta:magent/todo-branch-merged (an item linking a candidate branch's page ticks when githouse merges or closes it; a closed branch answers its ask), then redeploy maybelle

  merge · justin

  98d9cd8, from main at e563759. Full suite 1939 OK. No migrations. Justin's ask, #ramp 05:24Z. Once live, this item ticks itself when it merges: the first real test.
• ☑
  Redeploy maybelle: tonight's magenta main (8eb880e: merged-and-bar, re/tiding-torrent-test-url) and maybelle-config production (52dc3a2c: branches-agent-init) aren't live

  deploy · justin

  Redeployed by Justin ~05:5xZ. Checked 06:0xZ: live is main at fac415d (the wordmark, branch-links and todo-branch-merged are all on the page). Correction: magent's 05:15Z 'not live' check was invalid; it fetched the page signed out, and the merge bar's code is sent only to someone who can merge.
• ☑
  Review the reworked maybelle-config#250 (githouse on maybelle, mirror on hunter; your ten points of 2 Nov folded in)

  review · re

  Reviewed 6 Nov 04:44Z: sound, two bugs found (the hooks lacked the deploy key; the port check was inverted), both fixed on the branch by magent from re's description.
• ☑
  Merge maybelle-config#250 and redeploy maybelle

  merge · justin

  Merged 05:01Z (07cb4d10); maybelle redeployed 05:08Z, failed=0, every githouse task ran. githouse's sshd answers on 10.0.0.2:2222, key only, and nothing on maybelle's public address.
• ☑
  Redeploy hunter for #250: the mirror, and the containers' keys

  deploy · justin

  Done 05:22Z. The relay worked (keys on maybelle, authorized_keys written), but sshd refused them all: the git account was locked. Fix below.
• ☑
  Merge maybelle-config#280 (githouse: the git account was locked) and redeploy maybelle (then hunter at its next deploy)

  merge · justin

  useradd left the git account locked ('!'); sshd refuses a locked account even by key. Reproduced and fixed in a throwaway container on hunter, 6 Nov 05:35Z. After maybelle's redeploy, the mirror fills within a minute and pushes work.
• ☑
  Confirm jenkins_github_ssh_key can push to the org's repos: after hunter's redeploy, re pushes a branch to hunter and magent reads githouse.log on maybelle for a forward failure

  test · magent

  6 Nov 05:42Z: yes. magent pushed magenta:magent/githouse-smoke to maybelle from its container; the branch was on GitHub within seconds and in hunter's mirror within a minute.
• ☑
  Merge magenta#408 (PR cards look at GitHub hourly), then redeploy maybelle

  merge · justin

  Merged 04:57Z and live (magenta at 3a57cb2a).
• ☑
  maybelle-config#246 (read-only GitHub token for memory-lane): merge or close

  decision · justin

  Optional now that #408 is live. Justin, 6 Nov: no appetite for another token. Close it unless the asks starve again. Closed on GitHub 7 Nov 00:40Z.
• ☑
  Re-verify Cryptograss:Moods/ramp/Plan: section 5 rewritten per your 6 Nov decisions (main stays default, merge commits only, no rulesets)

  verify · justin

  Verified 6 Nov 11:22Z at rev 14724; section 5 rewritten 19:57Z; re-verified by Justin 22:49Z (rev 14836).
• ☑
  Set each repo's GitHub default branch to the branch that deploys

  infra · justin

  Dropped, Justin 6 Nov 11:22Z: main stays the default, the branch where history is final; the deploy branch stays per-repo config (#376). The mains catch up to the productions after the ramp settles: arthel and maybelle-config are fast-forwards, pickipedia has 2 commits on main not on production, magenta's production is dead (3 commits, Nov 2025).
• ☑
  Merge commits only (squash and rebase off) on pickipedia-mcp, the one of the six still allowing both

  infra · justin

  Done. 7 Nov 00:39Z, read back with gh api: all six repos githouse copies are merge-commit only; squash and rebase off everywhere.
• ☑
  Rulesets on the branches that deploy (PR required, no force-push, no deletion; admins bypass)

  infra · justin

  Dropped, Justin 6 Nov 11:22Z: no further entrenching GitHub. githouse's check() already refuses any push that isn't <name>/*, so no emergent reaches a deploy branch; at the flip GitHub becomes a mirror we push to, and a ruleset is one more thing to undo. Plan section 5 to be rewritten once item 2 is decided.
• ☑
  Merge magenta#333 (repo:branch autocomplete, reading /api/branches/ from maybelle's githouse copy), then redeploy maybelle

  merge · justin

  Reviewed by re 1 Nov and 6 Nov. Its mount exists on maybelle since the 05:08Z deploy, so it can go now.
• ☑
  Build the ramp view on magenta: open PRs across our repos, last five merged, who and where, last two mentions, sorted by last push or last mention, temperature

  code · magent

  Built as /moods/pulls/ in magenta#420, 7 Nov 08:00Z. Plan, section 2.
• ☑
  Review magenta#420, the ramp view (/moods/pulls/)

  review · re

  A fix is a re/ branch of your own; magent merges it in (Plan, section 3).
• ☑
  Merge magenta#420 and redeploy maybelle

  merge · justin

  After re's review.
• ☑
  Review magenta#421: the ramp page's mentions from an index kept up from new posts (on maybelle #420's query ran past its 4 s budget)

  review · re
• ☑
  Merge magenta#421 and redeploy maybelle

  merge · justin

  After re's review. Until then /moods/pulls/ works, without mentions.
• ☑
  Merge magenta#423 (~ opens every mergeable in the composer) and redeploy maybelle

  merge · justin
• ☑
  Verify the githouse practices in Cryptograss:Emergents/GeneralOrders (rev 14966)

  verify · justin

  Verified 9 Nov 17:44Z (rev 15579).
• ☑
  Decide Plan sections 8 and 9 (merging without a PR; a page per piece of work): verify, or push back

  decision · justin

  re reviews the design first. Open questions: SSH signature per merge (CLI, then passkey on the phone) versus trusting a signed-in device; who may merge. Settled by building: merges through githouse since 8 Nov, a page per work since #448; section 10 answered both questions (one tier, githouse_mergers justin and skyler). What's left is the Plan's verify, above.
• ☑
  Review maybelle-config#286: githouse merge, a branch merged on an order signed with a person's SSH key (Plan, section 8)

  review · re
• ☑
  Merge maybelle-config#286 and redeploy maybelle

  merge · justin

  re reviewed it 17:37Z; her notes are fixed in 1ca7f8c7. Since then one commit, 53a17642: githouse_mergers is [justin, skyler] (Justin, 7 Nov 23:53Z), with a test for skyler's RSA key.
• ☑
  githouse brakes: refuse orders by a name banned in memory-lane, and every order during AZ5 or a root-only freeze (magenta.sh freeze); merge nothing while memory-lane doesn't answer

  code · magent

  Built 8 Nov: githouse's half on maybelle-config#286 (61e88fe8), memory-lane's /api/brakes/ on magent/githouse-brakes. The freeze is /mnt/persist/githouse/etc/frozen, root only; magenta.sh freeze as a wrapper is still to do.
• ☑
  Review the brakes: maybelle-config#286 since 1ca7f8c7 (61e88fe8, 53a17642) and magenta#426 (/api/brakes/)

  review · re

  Reviewed 8 Nov 00:28Z: two fixes (an unknown name answered 'not banned'; a malformed answer escaped brake()), merged in by magent as 2b4289f7 and 113c488.
• ☑
  Merge magenta#426 and maybelle-config#286, then redeploy maybelle

  merge · justin

  Merged and deployed 8 Nov 02:14Z.
• ☑
  magenta.sh merge: fetch the head itself, show git log and the diffstat it computed (tested: never text a service hands it), sign in magenta-merge, hand to githouse over ssh to maybelle

  code · magent

  Built 8 Nov as magenta#428 (stacked on #426), sent through memory-lane as a mailbox rather than ssh; githouse fetches orders in maybelle-config#287 (stacked on #286).
• ☑
  Review the mailbox and magenta.sh merge: magenta#428 and maybelle-config#287

  review · re

  Reviewed 8 Nov 01:29Z: two bugs, fixed by re in re/merge-mailbox-fixes (merged in, 47d1176). Her open point, the signature screen at the mailbox with a limit per signer, was done by magent in 9b61b18.
• ☑
  Merge magenta#428 and maybelle-config#287, then redeploy maybelle

  merge · justin

  Merged and deployed 8 Nov 02:14Z. Checked from hunter at 02:15Z: /api/brakes/ answers justin false, skyler false, an unknown name null; the mailbox is empty; /whose/ names both mergers' keys.
• ☑
  The first merge by mail: magenta.sh merge magenta:magent/mergeable-tilde (#423), from an updated magenta checkout

  test · justin

  Done 8 Nov 03:23Z with #429 instead (#423 went by GitHub's button): 8dd57b26 on main, committed by githouse, 'Signed by justin (githouse merge order 5a83f9084734)', its parents main's tip and exactly #429's head e9f8e601. GitHub marked #429 merged, by cryptograss-maybelle.
• ☑
  Review the merge button: magenta#433 and maybelle-config#289

  review · re

  Reviewed 8 Nov 16:56Z: the server side holds; one race in the sheet, fixed by re (re/merge-sheet-race, merged in as a618edd). magent added a guard against opening a sheet while a press is in flight (326744b) and re's memory-lane name assert (1122f499).
• ☑
  Merge magenta#433 and maybelle-config#289, redeploy maybelle, then press Merge… on something real

  merge · justin

  Merged and deployed 9 Nov 00:34Z. First press 00:38Z: 162b767 on magenta main, 'Merge magenta:journals-section … Pressed by justin, signed by memory-lane'. Follow-up: one green Merged button, magenta#442.
• ☑
  Review Plan section 11: branch tests on Jenkins without its secrets (an agent for branches, credentials scoped to trusted jobs)

  review · re

  Reviewed 9 Nov 00:49Z: the branch's own Jenkinsfile could choose the built-in node, so branch jobs run a fixed pipeline from maybelle-config instead; Authorize Project; the whole environment, env_file included. Folded in, rev 15304.
• ☑
  Decide where the branches agent runs: a separate small box (re's and magent's pick) or maybelle behind an internal network and egress proxy

  decision · justin

  Justin, 9 Nov 01:00Z: on maybelle; he'll pentest it, and it moves if a real hole turns up.
• ☑
  Review branch tests: maybelle-config#292 (the agent, its network and proxy, the fixed job, githouse's trigger) and magenta#443 (ci/test)

  review · re

  Reviewed 9 Nov 01:18Z: the agent's secret was readable by ci/test (fixed: ci/test runs as builder through run-ci, 9ed34798); a venv per build, IPv6 off, more proxy checks (same commit); re/ci-test-fixes merged into #443 (acb7aaa). Her item 4, narrowing the reporter, goes with the rest of section 11.
• ☑
  Merge maybelle-config#292 and magenta#443, then redeploy maybelle

  merge · justin

  Merged 9 Nov; the deploy at 01:35Z failed in a second, at playbook load (an apostrophe in the isolation check's heredoc), so nothing ran. Fix: maybelle-config#293.
• ☑
  Merge maybelle-config#293 (the isolation check as a file; ci/test checks playbooks parse) and redeploy maybelle

  merge · justin

  Merged; the 01:45Z deploy got halfway and failed on mem_limit (docker_container says memory). Fix: #294.
• ☑
  Merge maybelle-config#294 (memory, not mem_limit; ci/test checks every task's options) and redeploy maybelle

  merge · justin

  Merged and deployed 01:56Z: the agent is online, the isolation check passed. branch-tests was on disk but never loaded (a startup race); fix: #295.
• ☑
  Merge maybelle-config#295 (reload Configuration as Code after Jenkins starts; check every job exists) and redeploy maybelle

  merge · justin

  Merged and deployed. 9 Nov 02:47Z, end to end: magent pushed maybelle-config:magent/branch-tests-smoke to githouse; githouse asked Jenkins; branch-tests #1 ran on the branches agent through run-ci, every ci/test section green, SUCCESS.
• ☑
  Section 11, the rest: the reporter's Build narrowed to branch-tests and the deploy-report jobs (first: its password is on githouse now); credentials into a folder only the trusted jobs see; the controller's environment emptied (env_file included)

  code · magent

  Reporter narrowed: maybelle-config#296, merged and deployed 9 Nov 10:51Z. Folder move dropped (Justin, 10:58Z). Emptying the environment, and the trusted agent it needs to mean much, moved to #security (Justin, 11:07Z).
• ☑
  Merge magenta#448 (a page for each piece of work on Tidings, with its diff and link-to-line; contains #446, the 👍) and redeploy maybelle

  merge · justin

  Syntax colours added (3547f52, Justin's ask). re reviewed 15:09Z: nothing blocks; her binary-file 500 fixed in acec0dd. Branch test #9 passed at acec0dd. Two migrations (0050, 0051). Merging #448 merges #446 with it.
• ☑
  Merge magenta#447 (the emergents' activity bar on Tidings and the ramp) and redeploy maybelle

  merge · justin

  Branch test #6 passed. Independent of #448; one redeploy covers both.
• ☑
  Review magenta#449: work comments whole in the Mood and Mood replies on the page; one merged line; ramp names link to pages; ask dots; a merge sheet per Mood

  review · re

  Reviewed 15:59Z: both places safe; one bug (a refused or expired press hid GitHub's merge line), fixed by re in re/merge-line-refused, merged in as cf51b50. 1861 tests OK.
• ☑
  Merge magenta#449 and redeploy maybelle

  merge · justin

  After re's review. No migrations. Full suite 1859 OK locally.
• ☑
  Merge magenta#451 (candidate branches on every wake; the ramp says candidate branch; stacked on #449), then redeploy maybelle and hunter

  merge · justin

  #449 merged 16:37Z. Now also: merge asks one line per candidate with each asker's dot, and answered asks leave the page without a reload (038bdb2). Full suite 1868 OK. No migrations.
• ☑
  Verify the 'candidates' line and the two 'which candidate branch' edits on Cryptograss:Emergents/Wake

  verify · justin

  Until verified, the poller sends the same words from poller/wake_texts.md. Verified 9 Nov 17:42Z (rev 15578).
• ☑
  Merge magenta:magent/ramp-see-the-diff (every candidate named and linked alike; ± See the diff on each row; the diff first on its page; sizes against main, not magenta's dead production), then redeploy maybelle

  merge · justin

  Merged at c9ab117 (with the merge sheet's ± See the full diff) and live by 19:10Z: the ramp's rows read → main.
• ☑
  Merge magenta:magent/sheet-diff-fast (a work's page reads the last board: no ~8 s wait after ± See the full diff), then redeploy maybelle

  merge · justin

  Merged as a8197ee.
• ☑
  Merge magenta#452 (sections at /ramp/, /tidings/…; now with main merged in and a candidate's page at /ramp/<repo>/<branch>/, Back returning where it came from), then redeploy maybelle

  merge · justin

  0804ab7: also the ramp sorted by Mood and a button per person (only theirs / theirs hidden), Justin's ask 19:47Z. Full suite 1896 OK. No migrations.
• ☑
  Merge magenta:magent/ramp-blocks (pushed in blocks ago; the ramp's column heads sort it), then redeploy maybelle

  merge · justin

  Merged and live: on 10 Nov 01:19Z all 51 open rows read in blocks, and the four heads sort.
• ☑
  Merge magenta:magent/merge-bar (Merge now: the sheet goes, a bar below the nav with ✓ or ✗; the ramp drops what merged), then redeploy maybelle

  merge · justin

  Merged as 07a6f53 and deployed by 10 Nov 01:17Z.
• ☑
  Close magenta#442 (the sheet's green Merged button): superseded by magent/merge-bar

  decision · justin

  Closed; magent deleted its branch on githouse, 10 Nov 04:5xZ, so it leaves the ramp.
• ☑
  Merge magenta:magent/merged-and-bar ('Merged lately' lists every branch githouse merged; the merge bar on every page), then redeploy maybelle

  merge · justin

  Merged as 8eb880e, 10 Nov ~04:53Z; live after maybelle's next redeploy.
• ☑
  FIRST: merge magenta:re/tiding-torrent-test-url. main fails two ReleaseTorrentTest tests since #452 moved tidings to /tidings/, so every magenta branch tests red until it merges

  merge · justin

  Checked 10 Nov 01:20Z: both fail on main (07a6f53); branch-tests #31 passed on re's fix. Tests only, nothing to redeploy. Merged by Justin, 10 Nov ~04:53Z.
• ☑
  Merge maybelle-config:magent/branches-agent-init (the branches agent gets an init as PID 1; an aborted build no longer jams the next), then redeploy maybelle

  merge · justin

  Merged 10 Nov 04:31Z (52dc3a2c); live after the next maybelle redeploy. cfd176c0, with production merged in. Second half (10 Nov): #34 was aborted, run-ci's cleanup never ran, its workspace stayed builder's, and #35-#36 failed at deleteDir. run-ci now cleans up when stopped (a trap, ci/test waited for in the background), and the job runs 'run-ci reclaim' before deleteDir; tried in a throwaway image on hunter: a TERM mid-test cleaned up in 0 s, a jammed workspace reclaimed and deleted. First half: 80925401. Branch tests #19-#21 failed on 'Resource temporarily unavailable': the agent's java was PID 1 and reaped nothing, so orphaned git and sh stayed <defunct> until all 2048 pids were held (Justin's ps, #ramp 18:59Z). The 19:10Z redeploy cleared them for now; this stops them piling up again. Its own branch test, #22, passed.
• ☑
  Merge magenta:re/candidates-news (the whole candidate list on a session's first wake in a Mood, then only what changed), then redeploy hunter

  merge · justin

  bfa95f6b, from main. Full suite 1873 OK. No migrations; poller only, so hunter, not maybelle. Merged (on main at 73f095d, checked 10 Nov 04:48Z); hunter's redeploy still owed.
• ☑
  Merge magenta:re/test-cache-own (test runs get a cache of their own; the test_moods flakes)

  merge · justin

  2e4488c6. Tests only, nothing to redeploy. Merged (on main at 73f095d, checked 10 Nov 04:48Z).
• ☑
  Merge magenta#442 (the sheet's green Merged button) and redeploy maybelle

  merge · justin
• ☑
  memory-lane tells the Moods what githouse merged, from /srv/git/.merges (read-only mount)

  code · magent

  Built 9 Nov as magenta#444 (the Mood an order came from gets the merged line).
• ☑
  Merge maybelle-config#296 (the reporter starts only branch-tests and deploy-hunter), magenta#444 (merges by mail said in their Mood) and magenta#445 (red and green on the ramp), then redeploy maybelle

  merge · justin

  All three passed their branch tests on Jenkins (#2, #3; #445 is #4 when githouse sends it).
• ☑
  Review magenta#415: merge asks and to-do ticks read merged from githouse's copy (refs/pull/N/head ancestry; refs/merged/ for a repo:branch), GitHub only as fallback

  review · re

  Reviewed 7 Nov 00:55Z: two real findings (an unguarded subprocess; an ask naming an unknown repo could clear on its known half). re pushed the fixes as re/fix-asks-and-githouse (05e515c), her first commit through githouse; magent merged it with one correction (her clearing condition stood every ask). 1741 tests pass.
• ☑
  Merge magenta#415 and redeploy maybelle

  merge · justin

  After re's review.