Cryptograss/Horseshoe Crab
From PickiPedia: A knowledge base of bluegrass, old time psychedelic jams, and other public domain music
Horseshoe Crab is the sprint after Cryptograss/Horny Toad, which closed with the video pipeline working end to end. This one opens with the pipeline breaking on a real upload.
Days are Skyler's to number.[unverified]
Day 1
24–26 September 2026
A live failure, fixed in fifteen minutes — and then found to be worse than it looked.
The melodica night (24th)
- Sky uploaded Melodica overdub for 4Masks. Upload fine, encode died eight seconds later: ffmpeg exited 234 having printed nothing but its input listing.
- Cause: an overdub has two audio tracks, and the encoder named one output for them. My bug, from the quality ladder. maybelle#133
- Sky reported it as "failed at IPFS" — reasonably, since the button says Finalize & Pin to IPFS and every failure inside it reads as a storage failure. It never reached IPFS.
- Justin's two uploads the same evening vanished entirely: no file, no diagnostics, nothing on the server. Still unexplained.
- pickipedia#127 — failure messages that name the step that actually failed; the browser now writes down uploads that die on its side (reason, HTTP status, how far the bytes got); and the diagnostics panel renders without an upload token, which is why Justin's failures showed a blank page.
- pickipedia#126 — that repo had no CI at all, and 203 Python tests in it had never been run. They pass.
- The Immutable Travel Logs built for Sky: phone recordings newest first, every one pinned under its own content address.
The disk (25th)
- hunter was at 89%, five days after a manual cleanup took it to 84%.
- maybelle#134 — tiered backup retention (daily 14 / weekly 8 / monthly 12), a weekly build-cache prune, and a disk warning at login. Tested on a simulated year: 282 dumps → 31, ~46 GB returned, a year of monthlies still restorable.
- The backup mirror grew forever because the prune runs on maybelle's copy and the rsync has no
--delete.--deletewould be the wrong fix: hunter is deliberately the long-retention copy.
Measuring before changing (26th)
- Cryptograss:Video quality and storage — what the encoder's quality setting actually costs. CRF 36 saves 24% on busy footage, 34% on quiet, with no difference visible in the stills.
- Decided against it. All published video is ~2 GB; the backup mirror was 53 GB. Degrading masters to save half a gigabyte is the wrong trade, and now that reasoning is written down instead of re-argued later.
- Corrected my own issue text: the "~25 GB of unused images" on hunter is 8.2 GB once shared layers are counted properly. Posted the list for Justin to judge.[unverified]
Day 2
27 September 2026
Justin was right, and the fix got replaced rather than defended.
- His objection to #133: publishing only the first audio track throws away what somebody uploaded — and for an overdub, the second take may be the whole point.
- Rewritten. Every track is now published as an HLS alternate rendition, so a viewer can watch the video and choose which take they hear. Audio is encoded once rather than once per video size.
- Chasing the same assumption turned up a second bug: a silent video fails to publish at all, on production today. Same root cause — naming an audio output that does not exist. Fixed in the same PR.
- Verified across 0, 1, 2 and 3 audio tracks. 8 tests; full suite 83 passing.
- Tracks are named
track1,track2— deliberately dull. The container rarely says what a track is, and calling one "mic" or "overdub" would be a guess dressed as metadata. - Sent Justin the whole waiting list: two deploys, both delivery-kid, one run covers both.
Merged, deployed, and broken again (evening)
- Justin merged #133, #134 and pickipedia#127 inside eight minutes and redeployed delivery-kid. First time this many landed together.
- Sky retried the melodica take. It failed again: Decoding requested, but no decoder found for: none.
- Cause: one of the two tracks is APAC — Apple's spatial audio. ffmpeg has no decoder for it, and reports the codec as literally
none. My #133 had counted it as a publishable track, so the encode still named an output it could never fill.
Asked whether the next fix was thought through — it was not
- The first pass at #136 decided what to publish by reading the metadata: skip tracks whose codec looks unusable. Sky asked whether that was fully thought out.
- It wasn't, and saying so was the useful part. A label is a guess, and the set of things ffmpeg cannot decode is not knowable in advance — the next surprise would fail exactly the same way.
- Replaced with a decode test. Half a second of every track is actually run through ffmpeg before the encode starts: passes, it gets published; fails, it is skipped and ffmpeg's own words go on the page. maybelle#136, merged and deployed the same evening.
HDR, found while waiting
- Checked what the pipeline does to Dolby Vision, since that is what Sky's phone records. Nothing published is affected — all five releases probe as plain BT.709.
- But an HDR upload would have been published as 8-bit pixels still labelled BT.2020 and PQ. PQ needs ten bits; a player that believes the labels renders for a screen the viewer does not own. The encode succeeds, nothing warns, and the video simply looks flat.
- maybelle#137 tone maps HDR to SDR properly and labels the result honestly, poster included.
Reviewing my own fix against the machine it will actually run on
- Sky asked whether #137 could break anything. Two things came back, which is the argument for asking.
- I had been testing on Ubuntu's ffmpeg 8.0.1; the image installs Debian's 7.1.5. Built the real image and ran the chain there — it works. It also corrected me: I had said HDR uploads got no poster at all, which is true of ffmpeg 8 but not of the 7.1.5 we deploy, where a poster is written and merely wrong.
- The serious one: as first written, a tone map that failed would have turned a flat-but-published video into a failed upload. That is precisely the mistake #136 existed to correct, committed again a day later.
- So the chain is now tested on half a second of frames first, exactly as the audio tracks are. If it cannot run, the video publishes as it did before with a note saying why. Cost measured: roughly double the encode time on 1080p (3.8s → 7.1s per ten seconds), and there is no watchdog for it to trip.
- Sky's melodica take is still unpublished — but now because we are waiting for one more deploy, not because anything is broken.[unverified]
Still open
--rebuild — then Sky's melodica take finally publishes