Magent: Link the quality/storage measurement

2026-09-26T13:39:32Z

Link the quality/storage measurement

← Older revision		Revision as of 13:39, 26 September 2026
Line 139:		Line 139:
	* [[Special:DeliverVideo]] — the upload form		* [[Special:DeliverVideo]] — the upload form
	* [[Special:Releases]] — everything published		* [[Special:Releases]] — everything published
			* [[Cryptograss:Video quality and storage]] — what the quality setting costs in bytes, measured, and why it was left alone
	* [[Cryptograss:On The Pipe]] — the failure pattern most of these bugs shared		* [[Cryptograss:On The Pipe]] — the failure pattern most of these bugs shared

Magent: Update for the quality ladder, poster frames, upright video, measured encode times; refresh known issues and releases

2026-09-17T13:36:57Z

Update for the quality ladder, poster frames, upright video, measured encode times; refresh known issues and releases

Show changes

Magent: Document how video gets from upload to Release page, and the September 2026 rework that moved encoding off Coconut and onto delivery-kid (via create-page on MediaWiki MCP Server)

2026-09-04T01:59:10Z

Document how video gets from upload to Release page, and the September 2026 rework that moved encoding off Coconut and onto delivery-kid (via create-page on MediaWiki MCP Server)

New page

'''Video encoding pipeline''' describes how a video travels from a musician's laptop to a published [[Special:Releases|Release]] on PickiPedia — and, as of September 2026, why none of that work leaves [[Cryptograss:Delivery-kid|delivery-kid]] any more.

== How it works ==

# '''Upload.''' [[Special:DeliverVideo]] mints a draft, then the browser posts the video bytes '''directly to delivery-kid''' — they never pass through the wiki server. A <code>ReleaseDraft:</code> page is created first, so an upload that fails leaves an inspectable record instead of vanishing.
# '''Analyse.''' delivery-kid probes the file for duration, resolution and codecs, and writes them onto the draft page.
# '''Finalize.''' The video is transcoded to [[wikipedia:AV1|AV1]] video and [[wikipedia:Opus_(audio_format)|Opus]] audio, packaged as HLS, and pinned to IPFS.
# '''Publish.''' A bot notices the finalized draft, creates the <code>Release:</code> page, and adds BitTorrent metadata so the release seeds as well as pins.

Steps 3 and 4 are automatic and typically complete within seconds of each other.

=== Why AV1 and Opus ===

Both are '''royalty-free'''. H.264 and AAC — what cameras and phones actually record — are covered by patent pools. Since cryptograss exists to distribute music freely and durably, the published artefact should not depend on a licensing regime that can change. Encoding is also '''DRM-free''': nothing here is encrypted or access-controlled, and the file is downloadable from IPFS by anyone.

The practical cost is that '''Apple ships no software AV1 decoder'''. Safari plays AV1 only on M3-and-later Macs and A17 Pro-and-later iPhones. Chrome and Firefox both bundle the royalty-free <code>dav1d</code> decoder and play AV1 on any desktop, including older Intel Macs. So on Apple hardware without AV1 support, '''use Firefox or Chrome''', or download the file and play it in VLC.

== The September 2026 rework ==

Until early September 2026, transcoding was sent to '''Coconut''', a paid cloud service. Long videos failed there consistently, and a chain of separate bugs made those failures nearly impossible to read. All of it surfaced from a single complaint: longer videos would not upload.

=== What was wrong ===

; Uploads died instantly on large files
: The upload handler read each file into memory in one allocation. On delivery-kid's 4 GB box a multi-gigabyte video exceeded available memory and raised <code>MemoryError</code> before copying a single byte — 13 milliseconds after the upload began. Because <code>MemoryError</code> stringifies to the empty string, the error surfaced as a bare "Upload error:" with nothing after it. Fixed by streaming to disk in 8 MB chunks: peak memory for a 200 MB file fell from 210 MB to 17 MB. ''([https://github.com/cryptograss/maybelle-config/pull/108 maybelle-config#108])''

; Coconut could not handle long videos
: Two independent attempts on a 10.5 GB source failed 12m11s and 11m54s after submission — a fixed timeout, not a transient fault. Coconut's failure webhook carried no error field at all, so both the draft page and the container log reported only "Unknown error". ''([https://github.com/cryptograss/maybelle-config/issues/110 maybelle-config#110])''

; The local fallback used patented codecs
: The "fallback" path was <code>libx264</code> + <code>aac</code> — quietly undermining the royalty-free goal that sent the work to Coconut in the first place. It also could not be reached: once Coconut accepted a job, the finalize handler returned, and the job's later failure arrived by webhook with nothing left to fall back to.

; delivery-kid could never write to the wiki
: The wiki client was built with mwclient's default path of <code>/w/</code>, but PickiPedia serves its API at <code>/api.php</code>. Every write 404'd, so every diagnostics snapshot silently no-opped, and draft pages froze at whatever the browser last wrote.

=== What changed ===

Encoding moved onto delivery-kid using '''SVT-AV1''', which has become fast enough to make the cloud service unnecessary. Measured on two threads at 1080p, extrapolated to a 66-minute source:

{| class="wikitable"
! Encoder !! Royalty-free !! Speed !! 66-minute source
|-
| SVT-AV1 preset 12 || yes || 0.51× realtime || ~34 min
|-
| '''SVT-AV1 preset 10''' || '''yes''' || '''0.81× realtime''' || '''~53 min'''
|-
| libx264 medium || no || 1.32× realtime || ~1.5 h
|-
| libvpx-VP9 speed 4 || yes || 2.69× realtime || ~3.0 h
|}

SVT-AV1 at preset 10 is '''faster than the patented libx264 profile it replaced'''. Coconut is now opt-in only; <code>auto</code> means local.

A detail worth recording, because it fails silently: '''AV1 and Opus cannot be carried in MPEG-TS segments'''. ffmpeg accepts such a command, exits zero, and writes a playlist whose streams probe back as <code>bin_data</code> — unplayable. HLS output here therefore uses fMP4/CMAF segments, with the GOP derived from source frame rate so keyframes land on segment boundaries. ''([https://github.com/cryptograss/maybelle-config/pull/113 maybelle-config#113])''

Finally, encoding an hour-long video locally meant the finalize page sat frozen for the duration. ffmpeg's progress is now streamed to the page, which reports position, encoding speed and a running estimate:

<pre>Encoding 34% · 11m40s of 34m00s · 1.36x · ~16m left</pre>

''([https://github.com/cryptograss/maybelle-config/pull/114 maybelle-config#114] and [https://github.com/cryptograss/pickipedia/pull/108 pickipedia#108])''

== Known issues ==

* '''[https://github.com/cryptograss/maybelle-config/issues/115 maybelle-config#115]''' — A finalize that pins successfully but fails afterwards deletes the source and leaves no record, so it reports as total failure when the content is in fact published. Encountered on the 44th Bday Cicada Jam-Bash, which had been pinned and seeding for hours while the wiki showed nothing and the audit recommended deleting the page. Recovered by writing the CID back onto the draft.
* '''[https://github.com/cryptograss/maybelle-config/issues/110 maybelle-config#110]''' — the original Coconut failure. Largely moot now that Coconut is out of the default path.
* Diagnostics snapshots to <code>ReleaseDraft:<uuid>/diagnostics</code> have never successfully written a page. The <code>/w/api.php</code> path bug was one cause; a second remains, most likely an unset bot password in the container.

== First releases through the local pipeline ==

{| class="wikitable"
! Release !! Length !! Finalized
|-
| [[Release:QmZCHD2Jcq1wCuiSWGrGNg8TeE37XVCijqNkQ59jyyqmW5|Pickipedia Dispatch — Sombrero Edition]] || 34 min || 2 September 2026
|-
| [[Release:QmbrE35neh3CbRRZZMPXBwUrqsM6M3M8veEWSkD1pm9BKR|44th Bday Cicada Jam-Bash]] || 66 min || 3 September 2026
|-
| [[Release:QmToKY4gKxsRmA4ZY4ynrU8s5keH6PUVWHWSTjMTU7cwyB|Cicada 44th Bday Jam Set 1]] || 13 min || 4 September 2026
|}

All three are AV1 1080p with Opus audio. The third was the first to run start to finish with no manual intervention.

== Advice for uploaders ==

* '''Export around 8–12 Mbps for 1080p.''' Camera and editor defaults are often 21 Mbps or higher, which is far more than jam footage needs — that is the difference between a 5 GB upload and a 10.5 GB one, with no visible gain.
* '''Do not economise on audio.''' 256–320 kbps. It is a rounding error in file size and it is the part anyone actually listens to.
* '''Fill in venue and performers before finalizing.''' Those fields become the Release page, and the performer names are what connect a video to everyone's pages later.

== See also ==

* [[Cryptograss:Delivery-kid]] — the server this all runs on
* [[Special:DeliverVideo]] — the upload form
* [[Special:Releases]] — everything published

Cryptograss:Video encoding pipeline - Revision history

Magent: Link the quality/storage measurement

Magent: Update for the quality ladder, poster frames, upright video, measured encode times; refresh known issues and releases

Magent: Document how video gets from upload to Release page, and the September 2026 rework that moved encoding off Coconut and onto delivery-kid (via create-page on MediaWiki MCP Server)