The Next Episode Would Not Play
Written later and filed to this date, as part of filling in every day of the year.
What. The video library on the box reads from a spinning drive in a USB enclosure on a dock. It developed a precise symptom: every episode already watched played, and the next one did not. Nothing in the server’s own logs said drive. Its library index, its already-cut stream segments and its poster art are all cached, so everything that had ever been touched kept working and only the untouched failed.
When it was noticed. Not as a dead drive, because it never looked like one. It looked like a few files that would not play, two days running. The answer was in the kernel log, which said the filesystem driver had failed to read the volume, and in the block device listing, which showed the drive under a different name than the mount table did.
Cause. The enclosure drops off the bus for a moment and comes back, and the kernel gives it the next free device node. The mount does not follow. It stays active on the node that no longer exists, so starting the mount unit again is a no-op (as far as the unit can tell, it is already mounted) and every uncached read returns an I/O error. A second failure hid behind the first: an enclosure that vanishes mid-write leaves the filesystem’s dirty flag set, and the driver then refuses to mount it at all, so detaching the dead mount alone left the drive unmounted, which looked like progress.
Fix. A healer script compares the mount’s source against the device the filesystem’s own identifier points to. When they differ it detaches the dead mount, lazily if something still holds it, runs the filesystem’s repair tool to reset the journal and clear the flag (the tool declines when the volume is too damaged, so a real fault still surfaces), and re-triggers the automount. The mount entry gained a setting that binds it to the device, so the next drop takes the mount down with it instead of leaving it lying. Thirty-five minutes after the first version the healer gained a lock: enabling its timer fired it at the same instant the installer was running it by hand, and the two raced.
What guards it now. The healer runs every two minutes and again whenever the enclosure re-plugs. It runs as root, so its exact bytes are pinned, and an edit that drifts from the pin raises an error on the status feed within a minute rather than failing quietly at the next drop. A check mode anyone can run says whether the mount and the device agree. And the server stopped hiding the one failure no software can fix: when another drive on the same dock came back reporting a capacity of zero bytes, viewers got a banner telling them to ask me, because the only remedy is a hand on the cable.