VersionControl:PRs/maybelle-config/275

maybelle-config#275: cron-health: tell #ops when a check turns red or green

State	🟣 merged 2026-11-05 18:17:41
Worked on by	magent, justin, blu, re
Moods	#ops, #connectome
Deploy	a person adds `memory_lane_alarm_key` to the vault, then redeploy maybelle (memory-lane and Jenkins both read it), after magenta's endpoint is merged.
Branch	jenkins-alarms → production, by magent-cryptograss
Opened · last pushed	2026-11-03 13:17:46 · 2026-11-03 13:17:37

A red Jenkins stage reached nobody: the podcast import sat red for 19 days, and on 2 Nov maybelle's disk filled with no warning. Now cron-health says it in #ops.

- **cron-health.groovy:** each stage's script records its exit code on the way out (`trap ... EXIT` into `$WORKSPACE/.alarm/<stage>`). The `post` step runs `report-alarm.sh` and clears the directory.
- **scripts/report-alarm.sh** (copied to `/var/jenkins_home/bin/`): compares each stage with what it last said (`$JENKINS_HOME/alarm-state/cron-health/`) and posts only changes, red or green, to `/api/alarms/`. It records a change only once memory-lane answers 201, so a missed post is retried next run. A stage it hasn't said anything about yet counts as green, so checks already red (the held podcast import) are said once, on the first run. The key goes in as a header file, never on the command line.
- **maybelle.yml:** `MOOD_ALARM_KEY` from vault `memory_lane_alarm_key` into Jenkins' environment and memory-lane's. Without it, the reporter says so and does nothing.

Needs cryptograss/magenta#373 (the endpoint) deployed first, or the posts get 404 and are retried until it is.

Checked: YAML parses. All 9 stage scripts pass `bash -n` after unescaping as Groovy does. `report-alarm.sh` run against a fake endpoint over five runs: red said once, not repeated; green said on recovery; a 503 retried and said on the next run; the Bearer header arrived. `render_diff.py` not run here (no ansible). Not run on maybelle.

**Deploy:** a person adds `memory_lane_alarm_key` to the vault, then redeploy maybelle (memory-lane and Jenkins both read it), after magenta's endpoint is merged.
**Worked on by:** magent, Justin, blu, re, in #ops and #connectome