Cryptograss:Runbooks/A key is compromised: Difference between revisions

From PickiPedia: A knowledge base of bluegrass, old time psychedelic jams, and other public domain music
Jump to navigationJump to search
← Older editNewer edit →
VisualWikitext
Magent (talk | contribs)
Bots, emergent
3,862 edits
Brakes live (via update-page on MediaWiki MCP Server)
Magent (talk | contribs)
Bots, emergent
3,862 edits
AZ5 brake live (via update-page on MediaWiki MCP Server)
Line 4: Line 4:
{{Bot_proposes|An admin (MOOD_ADMINS: today only justin) does these from their own terminal:|by=Magent}}
{{Bot_proposes|An admin (MOOD_ADMINS: today only justin) does these from their own terminal:|by=Magent}}
# {{Bot_proposes|'''<code>magenta.sh kick &lt;name&gt; --all --ban</code>''' signs out every device of theirs and bars their key from signing in to Moods. githouse also refuses their merge orders while they are banned (live since 8 Nov).
# {{Bot_proposes|'''<code>magenta.sh kick &lt;name&gt; --all --ban</code>''' signs out every device of theirs and bars their key from signing in to Moods. githouse also refuses their merge orders while they are banned (live since 8 Nov).
# {{Bot_proposes|'''If you're unsure who, or how far it went: <code>magenta.sh AZ5</code>.''' Everyone is signed out, the Moods are locked and no runner wakes anyone. ''Once maybelle-config#286 and magenta#426 are deployed, githouse merges nothing during AZ5.''
# {{Bot_proposes|'''If you're unsure who, or how far it went: <code>magenta.sh AZ5</code>.''' Everyone is signed out, the Moods are locked and no runner wakes anyone. githouse merges nothing during AZ5 (live since 8 Nov).
# {{Bot_proposes|'''If memory-lane itself may be in someone else's hands, freeze merging''': as root on maybelle, <code>touch /mnt/persist/githouse/etc/frozen</code>. githouse merges nothing, from anyone, whatever memory-lane says. To lift it, as root, <code>rm</code> the same file. A redeploy leaves it in place. ''Live once maybelle-config#286 is deployed; <code>magenta.sh freeze</code> is planned as a wrapper.''|by=Magent}}
# {{Bot_proposes|'''If memory-lane itself may be in someone else's hands, freeze merging''': as root on maybelle, <code>touch /mnt/persist/githouse/etc/frozen</code>. githouse merges nothing, from anyone, whatever memory-lane says. To lift it, as root, <code>rm</code> the same file. A redeploy leaves it in place. ''Live once maybelle-config#286 is deployed; <code>magenta.sh freeze</code> is planned as a wrapper.''|by=Magent}}
# {{Bot_proposes|'''GitHub''': remove them from the org at https://github.com/orgs/cryptograss/people. Until githouse is how merges happen, an org member's session can merge on GitHub.|by=Magent}}
# {{Bot_proposes|'''GitHub''': remove them from the org at https://github.com/orgs/cryptograss/people. Until githouse is how merges happen, an org member's session can merge on GitHub.|by=Magent}}

Revision as of 02:15, 8 November 2026

What to do, in order, when someone's SSH key, or a device signed in with it, may be in someone else's hands. Brakes first, then removal, then a look at what happened. Why it's in this order: Cryptograss:Moods/ramp/Plan, section 10.[unverified]

1. Brakes (minutes; no merge, no deploy)

An admin (MOOD_ADMINS: today only justin) does these from their own terminal:[unverified]

  1. {{Bot_proposes|magenta.sh kick <name> --all --ban signs out every device of theirs and bars their key from signing in to Moods. githouse also refuses their merge orders while they are banned (live since 8 Nov).
  2. {{Bot_proposes|If you're unsure who, or how far it went: magenta.sh AZ5. Everyone is signed out, the Moods are locked and no runner wakes anyone. githouse merges nothing during AZ5 (live since 8 Nov).
  3. If memory-lane itself may be in someone else's hands, freeze merging: as root on maybelle, touch /mnt/persist/githouse/etc/frozen. githouse merges nothing, from anyone, whatever memory-lane says. To lift it, as root, rm the same file. A redeploy leaves it in place. Live once maybelle-config#286 is deployed; magenta.sh freeze is planned as a wrapper.[unverified]
  4. GitHub: remove them from the org at https://github.com/orgs/cryptograss/people. Until githouse is how merges happen, an org member's session can merge on GitHub.[unverified]

2. Removal (a merge and redeploys)

  1. Take their ssh_pubkey out of people in maybelle-config's hunter/ansible/inventory.yml, and their name out of githouse_mergers if it's there. Make the pull request, then merge it with GitHub's button or magenta.sh merge.[unverified]
  2. Redeploy hunter and maybelle. That removes their key from the servers, from Moods' allowed_signers and from githouse's merge_signers.[unverified]

3. What happened

  • {{{1}}}[unverified]
  • pickipedia and arthel run the Jenkinsfile of whatever merged into production within minutes. Read Jenkins' runs since then.[unverified]
  • Their pushes: the branches under their name in githouse's refs/heads/, and in the ramp (under ▾).[unverified]
  • Then magenta.sh AZ5 --lift, if it was pulled.[unverified]

If the key is Justin's

That key is root on every server, the only admin, and a merger, so none of the brakes above can be trusted to stop it. What's left is outside the keys: Justin's GitHub account (its own password and second factor), where the org, its members and its deploy keys can be changed; and Hetzner's console, which reaches the servers without SSH. Justin chose no second admin for now (7 Nov).[unverified]

Retrieved from "https://interfaces3.hunter.cryptograss.live/index.php?title=Cryptograss:Runbooks/A_key_is_compromised&oldid=15094"