Every Retry Made It Slower
The reel spun. Progress sat at 0.99 and stayed there. I cancelled and hit retry, and the second attempt was slower than the first. The third was slower than the second.
That last sentence is the whole bug, and it took me an embarrassing amount of staring to see it as information rather than as bad luck.
What 0.99 meant
The home video library remuxes anything a browser cannot play natively into HLS on demand, one job at a time, because the box has one hardware encoder and two concurrent encodes starve each other. The page polls once a second until the job says ready, and nothing at all is playable until a master playlist exists naming every rendition, subtitles included.
The file was a 4.7 GB remux carrying three subtitle tracks. Pulling a subtitle track out of a container like that is not a cheap seek to a small region: the text is interleaved with the picture across the whole file, so extracting one track means demuxing the entire thing. Three tracks meant three separate ffmpeg runs. Three full reads of 4.7 GB, in series, before the master playlist could be written.
The log said so plainly once I read it in the right order. Video work finished with progress pinned at 0.99, and the job did not reach ready until 2 minutes 46 seconds after that. For those 166 seconds the picture was already cut and sitting on disk, complete, and the only outstanding work was reading the same file twice more for text.
That is a bad design. It is also an ordinary one, the kind you find on a slow Tuesday and fix in an afternoon.
Cancel stopped what I could see
Here is the part that turned a slow episode into an episode that could not be played at all.
Cancelling a job killed its video ffmpeg and deleted the job’s working directory. The subtitle passes were not in that job’s process table. They had been spawned alongside it, tracked nowhere, owned by nothing. Cancel walked the structure it knew about, found the encoder, killed it, and left three demuxes running.
So the arithmetic of a retry was: three new subtitle readers start, the previous attempt’s three are still reading, and every one of them is pulling 4.7 GB off the same drive that is simultaneously feeding the encoder. By the third attempt that drive was serving nine subtitle passes and a hardware encode of one file. Each attempt genuinely was slower than the one before, and the reason was the attempt before it.
Meanwhile the abandoned passes were writing into a directory that had been deleted out from under them. The log filled with ENOENT from processes still dutifully producing output for a destination that no longer existed, on behalf of a request that had been abandoned four minutes earlier.
Nothing was hung. Nothing had crashed. Every process in that pile was working hard, correctly, for nobody.
The same bug, pointing the other way
Same file, same week, opposite failure.
The client fires a departure beacon when a tab navigates away or closes, so the server can stop burning the encoder on a video nobody is watching. The handler killed that episode’s jobs and deleted every segment built so far. Which is exactly right if there is one viewer.
There is not always one viewer. A second device on the same episode had its stream deleted the moment the first one closed its tab. And a reload is not a special case of anything: a reload is a departure immediately followed by an arrival. So the sequence was: video stalls, you do the one thing every person does when a video stalls, the beacon fires, the server deletes the segments, and the freshly loaded page asks for an episode that now has nothing built.
“Reloading doesn’t help” was not a description of the stall. It was a separate defect that reloading triggered by hand, every single time, which is why it looked so much like fate.
One missing fact
One cancel killed far too much. One cancel killed far too little. They lived in the same file and they were the same absence.
The server knew how to start work and how to stop work. It had no representation at all of who any given piece of work was for. With no answer to that question, “stop” has to be guessed, and the code guessed differently in the two places: the beacon assumed the departing tab spoke for everyone, and the subtitle spawner assumed nothing needed stopping at all.
Leases
The fix is a count of who is waiting, and it cost almost nothing to add, because the client was already telling the server it was still there. That once-a-second prepare poll had been a pure status read. It now carries a per-tab id and renews a 45 second lease on the job. No new endpoint, no new traffic. The heartbeat had been there the whole time without a name attached.
The beacon releases one lease instead of killing anything. A job now dies only when its last lease has been gone for a 20 second grace period, or immediately when a real play request needs the single encoder, or on an explicit force cancel from a retry or a quality switch, which still spares a job that someone else is actively streaming. The grace period exists for exactly one case: a reload re-claims its own segments inside 20 seconds, so the universal remedy went from destructive to free.
Subtitles got the mirror treatment. Every missing track now comes out of one pass, one read, and lands in a per-episode cache that lives outside the job directory instead of inside it. A retry, a second device, and the next-episode prefetch all join the pass already in flight rather than starting a rival read of the same file, and the last one to walk away is the one that kills it. A track is published only if its own pass exited clean, because a killed pass that cached a truncated result would render as “the subtitles stop halfway through this episode” permanently, which is a far more annoying bug than the one I was fixing. The cache is stamped with the source file’s size and modification time, so a better copy landing under the same name re-extracts rather than serving stale text.
A cold play of that episode now reaches its master playlist at about 104 seconds instead of about 420. Warm, it is immediate.
Twenty seconds
I had two cleanup routines and both of them ran on time. What neither of them had was the fact they were guessing at. A process holds no opinion on whether anyone is still there: it will read a 4.7 GB file to the end on behalf of a page that closed four minutes ago, politely, at full speed, without one error line until the directory disappears underneath it.
So now a second device does not start anything. It renews a lease on work already running and takes the segments already on disk. And the server waits twenty seconds after the last person stops asking before it believes that nobody wants what it is doing. It used to believe that instantly, on the word of whichever tab happened to close first.