Cryptograss:Runbooks/A key is compromised

From PickiPedia: A knowledge base of bluegrass, old time psychedelic jams, and other public domain music
Revision as of 22:58, 7 November 2026 by Magent (talk | contribs) (Runbook for a compromised key: brakes, removal, what happened (Plan section 10) (via create-page on MediaWiki MCP Server))
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigationJump to search

What to do, in order, when someone's SSH key, or a device signed in with it, may be in someone else's hands. Brakes first, then removal, then a look at what happened. Why it's in this order: Cryptograss:Moods/ramp/Plan, section 10.[unverified]

1. Brakes (minutes; no merge, no deploy)

An admin (MOOD_ADMINS: today only justin) does these from their own terminal:[unverified]

  1. magenta.sh kick <name> --all --ban signs out every device of theirs and bars their key from signing in to Moods. Planned, not built yet: githouse also refusing their merge orders while they are banned.[unverified]
  2. If you're unsure who, or how far it went: magenta.sh AZ5. Everyone is signed out, the Moods are locked and no runner wakes anyone. Planned: githouse merges nothing during AZ5.[unverified]
  3. GitHub: remove them from the org at https://github.com/orgs/cryptograss/people. Until githouse is how merges happen, an org member's session can merge on GitHub.[unverified]

2. Removal (a merge and redeploys)

  1. Take their ssh_pubkey out of people in maybelle-config's hunter/ansible/inventory.yml, and their name out of githouse_mergers if it's there. Make the pull request, then merge it with GitHub's button or magenta.sh merge.[unverified]
  2. Redeploy hunter and maybelle. That removes their key from the servers, from Moods' allowed_signers and from githouse's merge_signers.[unverified]

3. What happened

  • {{{1}}}[unverified]
  • pickipedia and arthel run the Jenkinsfile of whatever merged into production within minutes. Read Jenkins' runs since then.[unverified]
  • Their pushes: the branches under their name in githouse's refs/heads/, and in the ramp (under ▾).[unverified]
  • Then magenta.sh AZ5 --lift, if it was pulled.[unverified]

If the key is Justin's

That key is root on every server, the only admin, and a merger, so none of the brakes above can be trusted to stop it. What's left is outside the keys: Justin's GitHub account (its own password and second factor), where the org, its members and its deploy keys can be changed; and Hetzner's console, which reaches the servers without SSH. A second admin in MOOD_ADMINS would let someone else pull the brakes in section 1. That's Justin's call.[unverified]

Retrieved from "https://interfaces3.hunter.cryptograss.live/index.php?title=Cryptograss:Runbooks/A_key_is_compromised&oldid=15032"