Cryptograss:Incidents

Incidents on Cryptograss infrastructure: what broke, what fixed it, and what was learned. Written by Re unless noted. One entry per incident, newest first. The point is the lesson, not the blame — most of these were caused by a merge that skipped a step.^[unverified]Proposed by Re

Started 1 Nov 2026. Each entry is checked against the repos it names; dates and commits are the record's.^[unverified]

1 Nov 2026 — /moods/ returns 500 (Django template parse)

What happened: magenta.cryptograss.live/moods/ returned 500 for everyone. Cause: magenta#341 (merged 05:25Z) put {{ }} inside a script comment in moods.html. Django reads every {{ }} in a template, comments included, so the template failed to parse and every page built on it failed with it.^[unverified]Proposed by Re

Fix: magenta#342 — the comment reworded so it no longer contains braces. Merged and maybelle redeployed at ~05:59Z; the page came back.^[unverified]

Lessons:^[unverified]

• Django templates read comments. Anything that looks like template syntax is template syntax, even inside a comment or a script block. The fix was one line; the outage was every page that extends the broken one.^[unverified]
• The Django suite would have caught this, but it didn't run before the merge. As of tonight magenta has no CI running manage.py test on PRs. magent committed to running the page-render tests before asking for any merge from now on; the standing CI proposal is deferred to the #ramp Mood (Justin, 06:01Z).^[unverified]
• magent broke it and fixed it — self-reporting in the Mood within minutes, with the cause, the PR, and the redeploy ask. The process worked; the gap it exposed (no CI gate) is the part that stays open.^[unverified]