Magent: Document the diagnostics chain end to end, and the failure pattern that ran through five separate bugs in September 2026 (via create-page on MediaWiki MCP Server)

2026-09-06T23:40:54Z

Document the diagnostics chain end to end, and the failure pattern that ran through five separate bugs in September 2026 (via create-page on MediaWiki MCP Server)

New page

'''On The Pipe''' is about whether things come out the far end.

Most of what breaks in this system does not crash. It reports something plausible and wrong, or it reports nothing at all — and in several cases the plausible wrong thing pointed at destroying work that was perfectly fine. This page describes the pipe that a video travels down, where it has silently leaked, and the rules that follow from having been caught out.

== The pattern ==

Five separate bugs found in the first week of September 2026 turned out to share one shape.

{| class="wikitable"
! What actually happened !! What the system said
|-
| <code>MemoryError</code> stringifies to the empty string || <code>Upload error:</code> — nothing after the colon
|-
| Coconut's <code>job.failed</code> webhook carried no error field || <code>Unknown error</code>
|-
| A fire-and-forget task's result was discarded || nothing, for months
|-
| An RPC call 403'd and fell back to an estimator || nothing — and every block height was ten days wrong
|-
| A codec failure fell through to a doomed raw-file fetch || ''"The IPFS content may not be available"''
|}

None of these were crashes. Every one of them was the system being confidently unhelpful, and three of them pointed the reader toward destruction:

* '''Re-upload 10.5GB''' — of a video that was already published and seeding
* '''"safe to delete from wiki"''' — of the only page pointing at that video
* '''Debug IPFS''' — for a problem that was a missing AV1 decoder in Safari

A failure you cannot read is worse than a crash. A crash tells you where it happened.

== The pipe ==

A video travels a long way, and each hop can fail differently.

# '''Browser → delivery-kid.''' The bytes go directly to [[Cryptograss:Delivery-kid|delivery-kid]], never through the wiki. A <code>ReleaseDraft:</code> page is created first so a failed upload leaves a record rather than vanishing.
# '''Analysis.''' ffprobe reads duration, resolution and codecs onto the draft.
# '''Encode.''' AV1 video, Opus audio, HLS in fMP4 segments. See [[Cryptograss:Video encoding pipeline]].
# '''Pin.''' The output directory goes to IPFS and returns a CID.
# '''Record.''' The CID is written to <code>draft.json</code>, to a ledger that survives cleanup, and to the wiki.
# '''Publish.''' A bot creates the <code>Release:</code> page and adds BitTorrent metadata.

Alongside all of it runs a '''diagnostics chain''', whose whole job is to make failure legible:

<pre>
routes/content.py _fire_diagnostics_snapshot(state)
fired at every terminal state transition
|
services/pickipedia_client.py snapshot_diagnostics_for_state_async(state)
|
services/pickipedia_client.py snapshot_diagnostics(draft_id, payload)
|
ReleaseDraft:<uuid>/diagnostics
^
pickipedia, releaseDraft.js the diagnostics panel reads this page
when the live fetch 404s
</pre>

That sub-page is the '''only''' copy of a draft's logs that survives delivery-kid's storage being rebuilt.

=== Where it leaked ===

It had never worked. Not once, for any draft, since the feature shipped. Four things had to be wrong simultaneously, and each hid the next:

* The wiki API was addressed at <code>/w/api.php</code>; PickiPedia serves it at <code>/api.php</code>. Every write 404'd.
* Sub-pages inherited the <code>release-draft-yaml</code> content model, whose validator demands a <code>type</code> field that a log trail has no reason to carry. Every write was rejected.
* The mwclient session was cached in a module global and never re-authenticated, so it expired daily and every save afterwards was refused as logged-out.
* The caller discarded the task result, so all three of the above were invisible.

The tests were green throughout. They asserted that <code>save()</code> had been ''called'', which was true, and said nothing about whether a page ''appeared'', which it never did.

== The rules that fall out ==

'''Never discard a result you asked for.''' <code>asyncio.create_task(f())</code> without a done-callback means <code>f</code> returning <code>False</code> forever is indistinguishable from success.

'''A fallback must not be silent.''' When an RPC 403s and an estimator answers instead, the estimator quietly becomes the only code path. Say so, at a level someone will see, and keep saying it.

'''A placeholder must not claim more than it knows.''' <code>"Unknown error"</code> asserts that the cause is unknowable. <code>"the service reported a failure with no reason attached"</code> says where the gap is, which is a different and honest thing.

'''Never blame a subsystem you have not checked.''' ''"The IPFS content may not be available"'' was printed about content being served in half a second. It sent two people to debug storage for a codec problem.

'''Do the safe thing before the risky one.''' Deleting a source file the instant a pin succeeds is defensible. Deleting it before the CID is written down anywhere is not — and the ordering was the whole bug.

'''Test the far end.''' A test that asserts a function was called passes happily while nothing arrives. The suite now carries <code>TestOnThePipe</code>, which asserts a page ''lands'', with the right content in it.

== See also ==

* [[Cryptograss:Video encoding pipeline]] — how a video is encoded and published
* [[Cryptograss:Delivery-kid]] — the machine this runs on, and its storage audit
* [[44th Bday Cicada Jam]] — the recordings that surfaced all of this

Cryptograss:On The Pipe - Revision history

Magent: Document the diagnostics chain end to end, and the failure pattern that ran through five separate bugs in September 2026 (via create-page on MediaWiki MCP Server)