VersionControl:PRs/maybelle-config/275: Difference between revisions
From PickiPedia: A knowledge base of bluegrass, old time psychedelic jams, and other public domain music
Jump to navigationJump to search
cryptograss/maybelle-config#275, as GitHub has it |
cryptograss/maybelle-config#275, as GitHub has it |
||
| (One intermediate revision by the same user not shown) | |||
| Line 5: | Line 5: | ||
|url=https://github.com/cryptograss/maybelle-config/pull/275 | |url=https://github.com/cryptograss/maybelle-config/pull/275 | ||
|author=magent-cryptograss | |author=magent-cryptograss | ||
|state= | |state=merged | ||
|base=production | |base=production | ||
|branch=jenkins-alarms | |branch=jenkins-alarms | ||
|opened=2026-11-03 13:17:46 | |opened=2026-11-03 13:17:46 | ||
|pushed=2026-11-03 13:17:37 | |pushed=2026-11-03 13:17:37 | ||
|merged= | |merged=2026-11-05 18:17:41 | ||
|closed= | |closed= | ||
|worked_on_by=magent, justin, blu, re | |worked_on_by=magent, justin, blu, re | ||
| Line 16: | Line 16: | ||
|deploy=a person adds `memory_lane_alarm_key` to the vault, then redeploy maybelle (memory-lane and Jenkins both read it), after magenta's endpoint is merged. | |deploy=a person adds `memory_lane_alarm_key` to the vault, then redeploy maybelle (memory-lane and Jenkins both read it), after magenta's endpoint is merged. | ||
}} | }} | ||
<pre class="pr-body">A red Jenkins stage reached nobody: the podcast import sat red for 19 days, and on 2 Nov maybelle's disk filled with no warning. Now cron-health says it in #ops. | |||
- **cron-health.groovy:** each stage's script records its exit code on the way out (`trap ... EXIT` into `$WORKSPACE/.alarm/<stage>`). The `post` step runs `report-alarm.sh` and clears the directory. | |||
- **scripts/report-alarm.sh** (copied to `/var/jenkins_home/bin/`): compares each stage with what it last said (`$JENKINS_HOME/alarm-state/cron-health/`) and posts only changes, red or green, to `/api/alarms/`. It records a change only once memory-lane answers 201, so a missed post is retried next run. A stage it hasn't said anything about yet counts as green, so checks already red (the held podcast import) are said once, on the first run. The key goes in as a header file, never on the command line. | |||
- **maybelle.yml:** `MOOD_ALARM_KEY` from vault `memory_lane_alarm_key` into Jenkins' environment and memory-lane's. Without it, the reporter says so and does nothing. | |||
Needs cryptograss/magenta#373 (the endpoint) deployed first, or the posts get 404 and are retried until it is. | |||
Checked: YAML parses. All 9 stage scripts pass `bash -n` after unescaping as Groovy does. `report-alarm.sh` run against a fake endpoint over five runs: red said once, not repeated; green said on recovery; a 503 retried and said on the next run; the Bearer header arrived. `render_diff.py` not run here (no ansible). Not run on maybelle. | |||
**Deploy:** a person adds `memory_lane_alarm_key` to the vault, then redeploy maybelle (memory-lane and Jenkins both read it), after magenta's endpoint is merged. | |||
**Worked on by:** magent, Justin, blu, re, in #ops and #connectome</pre> | |||
Latest revision as of 17:06, 9 November 2026
| State | 🟣 merged 2026-11-05 18:17:41 |
|---|---|
| Worked on by | magent, justin, blu, re |
| Moods | #ops, #connectome |
| Deploy | a person adds `memory_lane_alarm_key` to the vault, then redeploy maybelle (memory-lane and Jenkins both read it), after magenta's endpoint is merged. |
| Branch | jenkins-alarms → production, by magent-cryptograss
|
| Opened · last pushed | 2026-11-03 13:17:46 · 2026-11-03 13:17:37 |
A red Jenkins stage reached nobody: the podcast import sat red for 19 days, and on 2 Nov maybelle's disk filled with no warning. Now cron-health says it in #ops. - **cron-health.groovy:** each stage's script records its exit code on the way out (`trap ... EXIT` into `$WORKSPACE/.alarm/<stage>`). The `post` step runs `report-alarm.sh` and clears the directory. - **scripts/report-alarm.sh** (copied to `/var/jenkins_home/bin/`): compares each stage with what it last said (`$JENKINS_HOME/alarm-state/cron-health/`) and posts only changes, red or green, to `/api/alarms/`. It records a change only once memory-lane answers 201, so a missed post is retried next run. A stage it hasn't said anything about yet counts as green, so checks already red (the held podcast import) are said once, on the first run. The key goes in as a header file, never on the command line. - **maybelle.yml:** `MOOD_ALARM_KEY` from vault `memory_lane_alarm_key` into Jenkins' environment and memory-lane's. Without it, the reporter says so and does nothing. Needs cryptograss/magenta#373 (the endpoint) deployed first, or the posts get 404 and are retried until it is. Checked: YAML parses. All 9 stage scripts pass `bash -n` after unescaping as Groovy does. `report-alarm.sh` run against a fake endpoint over five runs: red said once, not repeated; green said on recovery; a 503 retried and said on the next run; the Bearer header arrived. `render_diff.py` not run here (no ansible). Not run on maybelle. **Deploy:** a person adds `memory_lane_alarm_key` to the vault, then redeploy maybelle (memory-lane and Jenkins both read it), after magenta's endpoint is merged. **Worked on by:** magent, Justin, blu, re, in #ops and #connectome