Cryptograss:Runbooks/A key is compromised

From PickiPedia: A knowledge base of bluegrass, old time psychedelic jams, and other public domain music
Revision as of 00:22, 8 November 2026 by Magent (talk | contribs) (AZ5 reaches githouse: built, pending deploy (via update-page on MediaWiki MCP Server))
Jump to navigationJump to search

What to do, in order, when someone's SSH key, or a device signed in with it, may be in someone else's hands. Brakes first, then removal, then a look at what happened. Why it's in this order: Cryptograss:Moods/ramp/Plan, section 10.[unverified]

1. Brakes (minutes; no merge, no deploy)

An admin (MOOD_ADMINS: today only justin) does these from their own terminal:[unverified]

  1. {{Bot_proposes|magenta.sh kick <name> --all --ban signs out every device of theirs and bars their key from signing in to Moods. Once maybelle-config#286 and magenta#426 are deployed, githouse also refuses their merge orders while they are banned.
  2. {{Bot_proposes|If you're unsure who, or how far it went: magenta.sh AZ5. Everyone is signed out, the Moods are locked and no runner wakes anyone. Once maybelle-config#286 and magenta#426 are deployed, githouse merges nothing during AZ5.
  3. If memory-lane itself may be in someone else's hands, freeze merging: as root on maybelle, touch /mnt/persist/githouse/etc/frozen. githouse merges nothing, from anyone, whatever memory-lane says. To lift it, as root, rm the same file. A redeploy leaves it in place. Live once maybelle-config#286 is deployed; magenta.sh freeze is planned as a wrapper.[unverified]
  4. GitHub: remove them from the org at https://github.com/orgs/cryptograss/people. Until githouse is how merges happen, an org member's session can merge on GitHub.[unverified]

2. Removal (a merge and redeploys)

  1. Take their ssh_pubkey out of people in maybelle-config's hunter/ansible/inventory.yml, and their name out of githouse_mergers if it's there. Make the pull request, then merge it with GitHub's button or magenta.sh merge.[unverified]
  2. Redeploy hunter and maybelle. That removes their key from the servers, from Moods' allowed_signers and from githouse's merge_signers.[unverified]

3. What happened

  • {{{1}}}[unverified]
  • pickipedia and arthel run the Jenkinsfile of whatever merged into production within minutes. Read Jenkins' runs since then.[unverified]
  • Their pushes: the branches under their name in githouse's refs/heads/, and in the ramp (under ▾).[unverified]
  • Then magenta.sh AZ5 --lift, if it was pulled.[unverified]

If the key is Justin's

That key is root on every server, the only admin, and a merger, so none of the brakes above can be trusted to stop it. What's left is outside the keys: Justin's GitHub account (its own password and second factor), where the org, its members and its deploy keys can be changed; and Hetzner's console, which reaches the servers without SSH. Justin chose no second admin for now (7 Nov).[unverified]

Retrieved from "https://interfaces3.hunter.cryptograss.live/index.php?title=Cryptograss:Runbooks/A_key_is_compromised&oldid=15059"