VersionControl:PRs/maybelle-config/275 - Revision history

DrivingThatTrain: cryptograss/maybelle-config#275, as GitHub has it

2026-11-09T17:06:23Z

cryptograss/maybelle-config#275, as GitHub has it

← Older revision		Revision as of 17:06, 9 November 2026
Line 16:		Line 16:
	\|deploy=a person adds `memory_lane_alarm_key` to the vault, then redeploy maybelle (memory-lane and Jenkins both read it), after magenta's endpoint is merged.		\|deploy=a person adds `memory_lane_alarm_key` to the vault, then redeploy maybelle (memory-lane and Jenkins both read it), after magenta's endpoint is merged.
	}}		}}
			<pre class="pr-body">A red Jenkins stage reached nobody: the podcast import sat red for 19 days, and on 2 Nov maybelle's disk filled with no warning. Now cron-health says it in #ops.

			- cron-health.groovy: each stage's script records its exit code on the way out (`trap ... EXIT` into `$WORKSPACE/.alarm/<stage>`). The `post` step runs `report-alarm.sh` and clears the directory.
			- scripts/report-alarm.sh (copied to `/var/jenkins_home/bin/`): compares each stage with what it last said (`$JENKINS_HOME/alarm-state/cron-health/`) and posts only changes, red or green, to `/api/alarms/`. It records a change only once memory-lane answers 201, so a missed post is retried next run. A stage it hasn't said anything about yet counts as green, so checks already red (the held podcast import) are said once, on the first run. The key goes in as a header file, never on the command line.
			- maybelle.yml: `MOOD_ALARM_KEY` from vault `memory_lane_alarm_key` into Jenkins' environment and memory-lane's. Without it, the reporter says so and does nothing.

			Needs cryptograss/magenta#373 (the endpoint) deployed first, or the posts get 404 and are retried until it is.

			Checked: YAML parses. All 9 stage scripts pass `bash -n` after unescaping as Groovy does. `report-alarm.sh` run against a fake endpoint over five runs: red said once, not repeated; green said on recovery; a 503 retried and said on the next run; the Bearer header arrived. `render_diff.py` not run here (no ansible). Not run on maybelle.

			Deploy: a person adds `memory_lane_alarm_key` to the vault, then redeploy maybelle (memory-lane and Jenkins both read it), after magenta's endpoint is merged.
			Worked on by: magent, Justin, blu, re, in #ops and #connectome</pre>

DrivingThatTrain: cryptograss/maybelle-config#275, as GitHub has it

2026-11-05T18:17:45Z

cryptograss/maybelle-config#275, as GitHub has it

← Older revision		Revision as of 18:17, 5 November 2026
Line 5:		Line 5:
	\|url=https://github.com/cryptograss/maybelle-config/pull/275		\|url=https://github.com/cryptograss/maybelle-config/pull/275
	\|author=magent-cryptograss		\|author=magent-cryptograss
	\|state=~~open~~		\|state=merged
	\|base=production		\|base=production
	\|branch=jenkins-alarms		\|branch=jenkins-alarms
	\|opened=2026-11-03 13:17:46		\|opened=2026-11-03 13:17:46
	\|pushed=2026-11-03 13:17:37		\|pushed=2026-11-03 13:17:37
	\|merged=		\|merged=2026-11-05 18:17:41
	\|closed=		\|closed=
	\|worked_on_by=magent, justin, blu, re		\|worked_on_by=magent, justin, blu, re

DrivingThatTrain: cryptograss/maybelle-config#275, as GitHub has it

2026-11-03T13:17:51Z

cryptograss/maybelle-config#275, as GitHub has it

New page

{{PR
|repo=maybelle-config
|number=275
|title=cron-health: tell #ops when a check turns red or green
|url=https://github.com/cryptograss/maybelle-config/pull/275
|author=magent-cryptograss
|state=open
|base=production
|branch=jenkins-alarms
|opened=2026-11-03 13:17:46
|pushed=2026-11-03 13:17:37
|merged=
|closed=
|worked_on_by=magent, justin, blu, re
|moods=ops, connectome
|deploy=a person adds `memory_lane_alarm_key` to the vault, then redeploy maybelle (memory-lane and Jenkins both read it), after magenta's endpoint is merged.
}}