Most of the Heat Had Already Exited
Written later and filed to this date, as part of filling in every day of the year.
The box was running hot enough to notice, and the health panel said ok. Memory 74 percent, ok. Swap 56 percent, ok. Load sat at ten on eight cores, and the number that should have alarmed someone was on no chart at all: the service that hosts the background agents had burned 45,900 CPU-seconds in a little over four hours, nearly three cores without a pause, while every process alive inside it accounted for about 2,100. The other 44,000 belonged to processes that had already exited.
Two rows built to miss it
The memory and swap rows are high-water marks by design. They say how full, not how busy. On most machines the two arrive together: pressure pushes free memory down, and something keyed to free memory notices. This box swaps into compressed RAM rather than to a disk, and there pressure never becomes a low free-memory number. It becomes compression work on the CPU: the kernel’s swap threads in the top ten, 256 MB swapped out in a five-second sample, and free memory parked at a figure no reaper would act on.
That blinded two things. The panel, and the agent runtime. A finished coding session keeps its worker warm for about an hour in case someone resumes it, and lets go sooner only when free memory falls under a threshold, which on compressed swap it never does. Sixteen worker processes held 5.9 GB. Eleven of them were sessions that had already finished, three of them more than a hundred minutes earlier, each still holding three or four hundred megabytes and a language server.
The rescue was the heat
Idle residue explains the memory. It does not explain 44,000 CPU-seconds of dead processes. For that, the watchdog’s journal had to be read beside the agent daemon’s log.
The watchdog exists so that background agents survive usage limits: when one is blocked, it waits, then brings the job back. Nine jobs that evening had hit a server rate limit instead, and the watchdog handled them the way it handles a spent allowance. Kill the job’s worker, dispatch a fresh one, send it a nudge. The daemon log paired every retry with the same two lines, a worker settled as killed and, two seconds later, a spare claimed. Each new worker started cold, re-read its transcript, hit the same limit and was killed again. Eight attempts each, nine jobs: 72 cold starts in 45 minutes, around 400 MB apiece.
The kill had been correct when it was written. The helper that brings a job back carries a comment saying it is safe because it is only ever called for a job whose worker is already dead. For a usage limit that was true; the worker has exited. A rate limit ends a turn and leaves the worker running. The same path, pointed at a live process, became a loop that killed it, paid to rebuild it, and walked it back into the wall.
What changed
The panel gained a row that reads the two signals meaning paging right now: the kernel’s memory pressure-stall share and the rate of swap traffic, classified ok, paging or stalled. Beside it, a loop retires finished sessions through the daemon’s own stop verb, and only the ones the daemon itself lists as done and idle, oldest first, capped, transcripts left resumable. Its first two passes retired sixteen sessions and handed back 1.5 GB of memory and 1.2 GB of swap. Its first pass under pressure also restarted the news aggregator mid-ingest for the swap that service held, and ten minutes later the restart rung was moved behind “stalled”: removing residue is free, and a restart costs a service its work in flight.
The other two changes lived in files no agent may edit, so they came to me as a patch to apply by hand. The fleet’s supervisor now retires a finished worker at the moment it collects the result, so the residue never forms, and holds new spawns while the stall share is at ten percent or more. The watchdog now asks whether a rate-limited worker is alive, and if it is, sends it a reply and nothing else.
None of the three blind spots was a bug where it stood. The rows measure what they say. The runtime’s threshold suits a machine that swaps to disk. The helper’s comment was true for every case its author had seen. Each was right about its own world and silent about the others: a swap that never frees memory, a limit that leaves the worker breathing.
One of the nine rescued jobs was the session that diagnosed all this. Its transcript opens with eight empty turns, one for each time the watchdog saved it.