VersionControl:PRs/maybelle-config/290: Difference between revisions
From PickiPedia: A knowledge base of bluegrass, old time psychedelic jams, and other public domain music
Jump to navigationJump to search
cryptograss/maybelle-config#290, as GitHub has it |
cryptograss/maybelle-config#290, as GitHub has it |
||
| Line 16: | Line 16: | ||
|deploy=redeploy delivery-kid | |deploy=redeploy delivery-kid | ||
}} | }} | ||
<pre class="pr-body">delivery-kid's BitTorrent seeder ran inside the web service's own process (since b6979d78, 15 Mar), so libtorrent's memory counted against uvicorn, and every restart or deploy of the web service stopped all seeding. On 8 Nov uvicorn was killed for running out of memory at 2.7 GB on the 4 GB CX22, in the middle of Skyler's finalize. | |||
**The seeder gets its own container** (re, `f620e219`) | |||
- `seeder` runs from the same image (`python -m app.seeder_main`), with `mem_limit: 768m`. Port 6881 moves to it. The web service no longer imports libtorrent, and a test enforces that. | |||
- The handoff goes through the seeding folder: `/enrich/torrent` writes into `seeding/.incoming/.tmp-*` and renames it into place, and the seeder picks it up within 5 s. `/torrent/status` reads the seeder's `.status.json`. | |||
- The deploy waits for the seeder to report itself running, and the 5-minute uptime check covers it too. | |||
- `/enrich/torrent` streams the IPFS archive to disk instead of reading it whole (`r.content`). The 1.4 GB subtitles Release has had no torrent since 2 Nov, and the Blue Railroad job asks for one every 2 minutes, so this is a strong suspect for the 2.7 GB. Not proven: watch `docker stats` after the deploy. | |||
- README rewritten to cover every container as this leaves them. | |||
**Slow work moved off the event loop** (magent, `e32c3a70`) | |||
- Unpacking the archive, hashing the pieces (`create_torrent`) and copying onto the Storage Box (`stage_for_seeding`) now run in threads, so a big Release no longer stalls every other request while its torrent is made. | |||
- With the loop free, the Blue Railroad job's next ask could start the same CID twice, so a CID that's already being made is refused ("Already being made") until it's done. | |||
- Tests: 146 pass, 6 skip, including `test_enrich_off_loop.py` (the loop keeps ticking while the hashing runs; one CID at a time). | |||
After merging: watch the subtitles Release get its torrent, and `docker stats` for `pinning` and `seeder`. | |||
Deploy: redeploy delivery-kid | |||
Worked on by: re, magent, Justin and Skyler, in #uploads-and-embeds</pre> | |||
Revision as of 17:06, 9 November 2026
| State | 🟢 open |
|---|---|
| Worked on by | re, magent, justin, skyler |
| Moods | #uploads-and-embeds |
| Deploy | redeploy delivery-kid |
| Branch | re/seeder-container → production, by magent-cryptograss
|
| Opened · last pushed | 2026-11-08 17:59:23 · 2026-11-08 17:58:57 |
delivery-kid's BitTorrent seeder ran inside the web service's own process (since b6979d78, 15 Mar), so libtorrent's memory counted against uvicorn, and every restart or deploy of the web service stopped all seeding. On 8 Nov uvicorn was killed for running out of memory at 2.7 GB on the 4 GB CX22, in the middle of Skyler's finalize.
**The seeder gets its own container** (re, `f620e219`)
- `seeder` runs from the same image (`python -m app.seeder_main`), with `mem_limit: 768m`. Port 6881 moves to it. The web service no longer imports libtorrent, and a test enforces that.
- The handoff goes through the seeding folder: `/enrich/torrent` writes into `seeding/.incoming/.tmp-*` and renames it into place, and the seeder picks it up within 5 s. `/torrent/status` reads the seeder's `.status.json`.
- The deploy waits for the seeder to report itself running, and the 5-minute uptime check covers it too.
- `/enrich/torrent` streams the IPFS archive to disk instead of reading it whole (`r.content`). The 1.4 GB subtitles Release has had no torrent since 2 Nov, and the Blue Railroad job asks for one every 2 minutes, so this is a strong suspect for the 2.7 GB. Not proven: watch `docker stats` after the deploy.
- README rewritten to cover every container as this leaves them.
**Slow work moved off the event loop** (magent, `e32c3a70`)
- Unpacking the archive, hashing the pieces (`create_torrent`) and copying onto the Storage Box (`stage_for_seeding`) now run in threads, so a big Release no longer stalls every other request while its torrent is made.
- With the loop free, the Blue Railroad job's next ask could start the same CID twice, so a CID that's already being made is refused ("Already being made") until it's done.
- Tests: 146 pass, 6 skip, including `test_enrich_off_loop.py` (the loop keeps ticking while the hashing runs; one CID at a time).
After merging: watch the subtitles Release get its torrent, and `docker stats` for `pinning` and `seeder`.
Deploy: redeploy delivery-kid
Worked on by: re, magent, Justin and Skyler, in #uploads-and-embeds