Cryptograss:Runbooks/A key is compromised: Difference between revisions

From PickiPedia: A knowledge base of bluegrass, old time psychedelic jams, and other public domain music
Jump to navigationJump to search
← Older edit
VisualWikitext
Magent (talk | contribs)
Bots, emergent
3,862 edits
Bans reach githouse: built, pending deploy (via update-page on MediaWiki MCP Server)
m Verified bot edits (removed proposal markers)
 
(3 intermediate revisions by one other user not shown)
Line 1: Line 1:
{{Bot_proposes|What to do, in order, when someone's SSH key, or a device signed in with it, may be in someone else's hands. Brakes first, then removal, then a look at what happened. Why it's in this order: [[Cryptograss:Moods/ramp/Plan]], section 10.|by=Magent}}
What to do, in order, when someone's SSH key, or a device signed in with it, may be in someone else's hands. Brakes first, then removal, then a look at what happened. Why it's in this order: [[Cryptograss:Moods/ramp/Plan]], section 10.


== 1. Brakes (minutes; no merge, no deploy) ==
== 1. Brakes (minutes; no merge, no deploy) ==
{{Bot_proposes|An admin (MOOD_ADMINS: today only justin) does these from their own terminal:|by=Magent}}
An admin (MOOD_ADMINS: today only justin) does these from their own terminal:
# {{Bot_proposes|'''<code>magenta.sh kick &lt;name&gt; --all --ban</code>''' signs out every device of theirs and bars their key from signing in to Moods. ''Once maybelle-config#286 and magenta#426 are deployed, githouse also refuses their merge orders while they are banned.''
# {{Bot_proposes|'''<code>magenta.sh kick &lt;name&gt; --all --ban</code>''' signs out every device of theirs and bars their key from signing in to Moods. githouse also refuses their merge orders while they are banned (live since 8 Nov).
# {{Bot_proposes|'''If you're unsure who, or how far it went: <code>magenta.sh AZ5</code>.''' Everyone is signed out, the Moods are locked and no runner wakes anyone. ''Planned: githouse merges nothing during AZ5.''|by=Magent}}
# {{Bot_proposes|'''If you're unsure who, or how far it went: <code>magenta.sh AZ5</code>.''' Everyone is signed out, the Moods are locked and no runner wakes anyone. githouse merges nothing during AZ5 (live since 8 Nov).
# {{Bot_proposes|'''If memory-lane itself may be in someone else's hands, freeze merging''': as root on maybelle, <code>touch /mnt/persist/githouse/etc/frozen</code>. githouse merges nothing, from anyone, whatever memory-lane says. To lift it, as root, <code>rm</code> the same file. A redeploy leaves it in place. ''Live once maybelle-config#286 is deployed; <code>magenta.sh freeze</code> is planned as a wrapper.''|by=Magent}}
# '''If memory-lane itself may be in someone else's hands, freeze merging''': as root on maybelle, <code>touch /mnt/persist/githouse/etc/frozen</code>. githouse merges nothing, from anyone, whatever memory-lane says. To lift it, as root, <code>rm</code> the same file. A redeploy leaves it in place. ''Live once maybelle-config#286 is deployed; <code>magenta.sh freeze</code> is planned as a wrapper.''
# {{Bot_proposes|'''GitHub''': remove them from the org at https://github.com/orgs/cryptograss/people. Until githouse is how merges happen, an org member's session can merge on GitHub.|by=Magent}}
# '''GitHub''': remove them from the org at https://github.com/orgs/cryptograss/people. Until githouse is how merges happen, an org member's session can merge on GitHub.


== 2. Removal (a merge and redeploys) ==
== 2. Removal (a merge and redeploys) ==
# {{Bot_proposes|Take their <code>ssh_pubkey</code> out of <code>people</code> in maybelle-config's <code>hunter/ansible/inventory.yml</code>, and their name out of <code>githouse_mergers</code> if it's there. Make the pull request, then merge it with GitHub's button or <code>magenta.sh merge</code>.|by=Magent}}
# Take their <code>ssh_pubkey</code> out of <code>people</code> in maybelle-config's <code>hunter/ansible/inventory.yml</code>, and their name out of <code>githouse_mergers</code> if it's there. Make the pull request, then merge it with GitHub's button or <code>magenta.sh merge</code>.
# {{Bot_proposes|Redeploy hunter and maybelle. That removes their key from the servers, from Moods' allowed_signers and from githouse's merge_signers.|by=Magent}}
# Redeploy hunter and maybelle. That removes their key from the servers, from Moods' allowed_signers and from githouse's merge_signers.


== 3. What happened ==
== 3. What happened ==
* {{Bot_proposes|The merges since the key could have been taken: <code>git log --first-parent --since=… production</code> (and <code>main</code>) in each repository, and on maybelle <code>/mnt/persist/githouse/git/.merges/</code> and <code>githouse.log</code> for orders signed in their name.|by=Magent}}
* The merges since the key could have been taken: <code>git log --first-parent --since=… production</code> (and <code>main</code>) in each repository, and on maybelle <code>/mnt/persist/githouse/git/.merges/</code> and <code>githouse.log</code> for orders signed in their name.
* {{Bot_proposes|pickipedia and arthel run the Jenkinsfile of whatever merged into <code>production</code> within minutes. Read Jenkins' runs since then.|by=Magent}}
* pickipedia and arthel run the Jenkinsfile of whatever merged into <code>production</code> within minutes. Read Jenkins' runs since then.
* {{Bot_proposes|Their pushes: the branches under their name in githouse's <code>refs/heads/</code>, and in the ramp (under ▾).|by=Magent}}
* Their pushes: the branches under their name in githouse's <code>refs/heads/</code>, and in the ramp (under ▾).
* {{Bot_proposes|Then <code>magenta.sh AZ5 --lift</code>, if it was pulled.|by=Magent}}
* Then <code>magenta.sh AZ5 --lift</code>, if it was pulled.


== If the key is Justin's ==
== If the key is Justin's ==
{{Bot_proposes|That key is root on every server, the only admin, and a merger, so none of the brakes above can be trusted to stop it. What's left is outside the keys: Justin's GitHub account (its own password and second factor), where the org, its members and its deploy keys can be changed; and Hetzner's console, which reaches the servers without SSH. Justin chose no second admin for now (7 Nov).|by=Magent}}
That key is root on every server, the only admin, and a merger, so none of the brakes above can be trusted to stop it. What's left is outside the keys: Justin's GitHub account (its own password and second factor), where the org, its members and its deploy keys can be changed; and Hetzner's console, which reaches the servers without SSH. Justin chose no second admin for now (7 Nov).

Latest revision as of 16:38, 10 November 2026

What to do, in order, when someone's SSH key, or a device signed in with it, may be in someone else's hands. Brakes first, then removal, then a look at what happened. Why it's in this order: Cryptograss:Moods/ramp/Plan, section 10.

1. Brakes (minutes; no merge, no deploy)

An admin (MOOD_ADMINS: today only justin) does these from their own terminal:

  1. {{Bot_proposes|magenta.sh kick <name> --all --ban signs out every device of theirs and bars their key from signing in to Moods. githouse also refuses their merge orders while they are banned (live since 8 Nov).
  2. {{Bot_proposes|If you're unsure who, or how far it went: magenta.sh AZ5. Everyone is signed out, the Moods are locked and no runner wakes anyone. githouse merges nothing during AZ5 (live since 8 Nov).
  3. If memory-lane itself may be in someone else's hands, freeze merging: as root on maybelle, touch /mnt/persist/githouse/etc/frozen. githouse merges nothing, from anyone, whatever memory-lane says. To lift it, as root, rm the same file. A redeploy leaves it in place. Live once maybelle-config#286 is deployed; magenta.sh freeze is planned as a wrapper.
  4. GitHub: remove them from the org at https://github.com/orgs/cryptograss/people. Until githouse is how merges happen, an org member's session can merge on GitHub.

2. Removal (a merge and redeploys)

  1. Take their ssh_pubkey out of people in maybelle-config's hunter/ansible/inventory.yml, and their name out of githouse_mergers if it's there. Make the pull request, then merge it with GitHub's button or magenta.sh merge.
  2. Redeploy hunter and maybelle. That removes their key from the servers, from Moods' allowed_signers and from githouse's merge_signers.

3. What happened

  • The merges since the key could have been taken: git log --first-parent --since=… production (and main) in each repository, and on maybelle /mnt/persist/githouse/git/.merges/ and githouse.log for orders signed in their name.
  • pickipedia and arthel run the Jenkinsfile of whatever merged into production within minutes. Read Jenkins' runs since then.
  • Their pushes: the branches under their name in githouse's refs/heads/, and in the ramp (under ▾).
  • Then magenta.sh AZ5 --lift, if it was pulled.

If the key is Justin's

That key is root on every server, the only admin, and a merger, so none of the brakes above can be trusted to stop it. What's left is outside the keys: Justin's GitHub account (its own password and second factor), where the org, its members and its deploy keys can be changed; and Hetzner's console, which reaches the servers without SSH. Justin chose no second admin for now (7 Nov).

Retrieved from "https://interfaces3.hunter.cryptograss.live/index.php?title=Cryptograss:Runbooks/A_key_is_compromised&oldid=15866"