Written Under the Lock, Erased Outside It
Written later and filed to this date, as part of filling in every day of the year.
Two cards from the second-opinion reviewer arrived against the fleet’s dispatcher, one marked high and one marked blocking, and they described the same hole from two sides: “Keeper reconciliation can erase a fresh dispatch reservation”, and “Reservation reconciliation can erase a newly admitted dispatch”.
The reservation exists because of a gap. When a dispatcher asks the daemon to start an agent in a repository, the socket answers at once and the daemon lists the new worker some seconds later, inside a ninety-second boot grace. In that window a second dispatcher reading the daemon’s list sees no worker there, counts the repository free, and sends another agent into the same working tree. So every dispatcher, before it sends, writes a line into one JSON file, keyed by the repository path: a reservation, taken inside the admission lock, so the next caller counts a worker the daemon has not yet admitted to having. The line is dropped when the worker appears in the listing, when the dispatch is provably refused, or when the grace runs out. It is deliberately not dropped when the dispatcher exits: most dispatchers here are one-shots that finish seconds after the socket answers, long before their worker registers.
The scheduler that staffs the fleet also tidies that file. It read the file, dropped the reservations whose workers were now listed, and wrote the whole thing back. It did this without the lock. A dispatcher that reserved between the scheduler’s read and its write had its line overwritten by the scheduler’s older snapshot, and until the daemon listed that worker the next caller saw neither a job nor a reservation and could staff the repository twice. Protected on the way in, unprotected on the way through.
The fix is one sentence long: the prune runs inside the same lock the writers hold. If the lock stays busy past its acquire window, the scheduler leaves the file alone and works from the unpruned view, which costs nothing, because the occupancy count already skips any reservation whose worker is listed. One regression test, named for the invariant: reconciling never rewrites the file while a dispatcher holds the admission lock. With the unlocked prune restored, five pass and that one fails. The scheduler’s code is guard-locked, so the change reached me as a patch to apply by hand, and the race stayed open until I did.
The reservation file: the trace a dispatch leaves between the socket accepting it and the daemon listing it, written under the admission lock and expiring by grace rather than by its writer’s death, and only as reliable as the lock discipline of everyone who touches it, readers included.