Nothing Was Saved Until Everything Was
What happened. For three days in October my news aggregator’s pipeline did not complete a run. Fifty-seven runs started and fifty-seven were interrupted. The site stayed up the whole time; a run in progress is a normal state, and a run was always in progress.
When I noticed. On the third day, watching a run live with the service journal open. Every run died about thirty minutes in, in the same place: stage eight, the pass that scores every article in the window against the signal engine.
The cause. Three things above the pipeline were killing it, and two things inside it turned each death into a loop. The stage reached 3.6 GB seven minutes in, on a box that was already paging, and twice the kernel’s out-of-memory killer took it; the time I watched, the trigger was an unrelated build with swap full. The other fifty-five deaths were my own sentinel. It probes the service, and its ladder declares a five-minute grace before any remedy, which the remedy path had never read, so one failed probe during a page storm restarted the backend at once.
None of that costs three days on its own. The first property inside the pipeline: a stop request had nowhere to land. The stages checked nothing between articles, and each wrapped its work in a guard that catches ordinary exceptions, so a stop raised as one would have been swallowed too. Every shutdown hung until the service manager killed it outright. The second: stage eight wrote nothing until it had finished. Scores for thousands of articles sat in memory and committed as one batch at the end. A run killed at minute thirty had done thirty minutes of work and saved none of it. The next run began at zero, after the hour of backoff the pipeline charges itself for a run that died.
The fix. Three changes, the same evening. The stop flag is now process-wide and raised as a base exception, the kind the stages’ guards cannot catch, checked at every stage boundary and between worker batches; a run stopped that way is recorded as stopped cleanly, and the next process catches up at once instead of serving the hour. Stage eight loads, scores and commits one page of articles at a time, logs its memory after each page, keeps only the running top of the list for the expensive pass that follows, and persists whatever that pass finished when a stop cuts it short. And the sentinel honours the grace its own ladder declared.
What guards it. Two test files pin the stop path and the page-by-page commits; the backend suite stands at 1,562. The memory ceiling and the swap pressure on the box are unchanged, on purpose: the next run names its growth per page in the journal, and I would rather read that line than guess at it.
The part I keep is the second property. A stage that saves only at the end has an all-or-nothing contract with every supervisor above it, and there were three supervisors above this one, none of them told. The kernel, the service manager and my own sentinel each did something reasonable, and the stage answered every one of them by starting over.