VersionControl:PRs/maybelle-config/275
From PickiPedia: A knowledge base of bluegrass, old time psychedelic jams, and other public domain music
Jump to navigationJump to search
| State | 🟣 merged 2026-11-05 18:17:41 |
|---|---|
| Worked on by | magent, justin, blu, re |
| Moods | #ops, #connectome |
| Deploy | a person adds `memory_lane_alarm_key` to the vault, then redeploy maybelle (memory-lane and Jenkins both read it), after magenta's endpoint is merged. |
| Branch | jenkins-alarms → production, by magent-cryptograss
|
| Opened · last pushed | 2026-11-03 13:17:46 · 2026-11-03 13:17:37 |
A red Jenkins stage reached nobody: the podcast import sat red for 19 days, and on 2 Nov maybelle's disk filled with no warning. Now cron-health says it in #ops. - **cron-health.groovy:** each stage's script records its exit code on the way out (`trap ... EXIT` into `$WORKSPACE/.alarm/<stage>`). The `post` step runs `report-alarm.sh` and clears the directory. - **scripts/report-alarm.sh** (copied to `/var/jenkins_home/bin/`): compares each stage with what it last said (`$JENKINS_HOME/alarm-state/cron-health/`) and posts only changes, red or green, to `/api/alarms/`. It records a change only once memory-lane answers 201, so a missed post is retried next run. A stage it hasn't said anything about yet counts as green, so checks already red (the held podcast import) are said once, on the first run. The key goes in as a header file, never on the command line. - **maybelle.yml:** `MOOD_ALARM_KEY` from vault `memory_lane_alarm_key` into Jenkins' environment and memory-lane's. Without it, the reporter says so and does nothing. Needs cryptograss/magenta#373 (the endpoint) deployed first, or the posts get 404 and are retried until it is. Checked: YAML parses. All 9 stage scripts pass `bash -n` after unescaping as Groovy does. `report-alarm.sh` run against a fake endpoint over five runs: red said once, not repeated; green said on recovery; a 503 retried and said on the next run; the Bearer header arrived. `render_diff.py` not run here (no ansible). Not run on maybelle. **Deploy:** a person adds `memory_lane_alarm_key` to the vault, then redeploy maybelle (memory-lane and Jenkins both read it), after magenta's endpoint is merged. **Worked on by:** magent, Justin, blu, re, in #ops and #connectome