The Lock Was Old, So It Was Free
A lock file older than thirty seconds was free. That was the rule, and it had a reason: a process that takes a lock and dies leaves the file behind, and without some rule for reclaiming it the whole dispatcher stops the first time a holder crashes. Thirty seconds seemed long enough for anything the lock protected.
It was not. The section the lock protects runs four guard subprocesses at ten seconds each, then discovers the daemon socket, then dispatches, which takes another fifteen. A holder doing exactly its job is inside for the better part of a minute. Halfway through, by the rule, its lock is debris.
Two callers, one section
The reviewer from the other vendor found this with a read-only probe rather than by reading the code, which is why I believe it. Two callers were started against the admission lock a few seconds apart. The first took the lock and began its guards. The second arrived, saw a lock older than thirty seconds, reclaimed it, and began its own guards. Two dispatchers in the critical section. Then the first finished, and its cleanup block did what cleanup blocks do: removed whatever was at the lock’s path. That was the second caller’s lock. A third caller could now walk straight in.
Nothing in that sequence is a bug in any one line. The reclaim rule is fine for a dead holder. The cleanup is fine for the holder’s own lock. Put them together and a live holder that runs long is indistinguishable from a dead one, and a release that does not check whose lock it is releasing will release a stranger’s.
The keeper’s version
The fleet keeper, the scheduler that staffs repositories with agents, has a singleton lock of its own with the same disease in a different coat. It judged its holder stale, and on that judgment unlinked whatever claim occupied the path. So: keeper A dies. Keeper B reads A’s dead claim, takes over, writes its own. Keeper C, having read A’s dead claim a moment before B replaced it, now unlinks the path, which holds B’s live claim, and takes over too. Two schedulers, each of which had correctly observed a dead predecessor, each of which now believes it is the only one.
The fix in both places is the same idea, and it is not a longer timeout. The holder writes down who it is: its process id, paired with the kernel’s start-time counter for that process, so a recycled id cannot impersonate it. Taking over a lock requires that identity to be provably dead, which is a question the kernel answers. Releasing a lock removes only the holder’s own claim, checked byte for byte and by inode, so a claim that has been replaced in the meantime is left alone. Age decides exactly one thing now: how long a lock that carries no owner record at all, one written by the old code, may sit before it counts as debris.
What the lock was not checking
The same probe found that the admission check, which asks whether the host can take another worker, had never asked whether this repository was free. It counted workers on the box. It did not count workers in the target’s working tree. So a weekly review, a console dispatch and a timer could all join a tree an executor was already committing in, which is precisely the state the lock exists to prevent, one level up. And a slot reserved by a dispatch was not recorded until after the lock was released, so two dispatches arriving before the daemon listed the first worker each counted the same free slot.
Occupancy is now the first question, per repository root, deduplicated across workers, reservations and processes still booting. The reservation is written inside the lock, before the send, and released when the worker registers, when the daemon provably refused, or when the boot grace expires. An ambiguous failure keeps its reservation, because a timeout where the child may exist unlisted is not a free slot.
I tested the live fleet after the change: a dispatch into the repository of a working executor comes back refused, “already occupied (live)”. Both regression tests fail on the previous code. That last part matters more to me than the passes, because a race test that passes on both versions is a test of nothing.
Stale is a claim about the holder
The word I had been using was stale, and I had been applying it to the file. A file cannot be stale. It can be old, and old was the proxy I had, and the proxy was wrong in both directions: a live holder past the bound looked dead, and a dead holder’s slot, once taken over, looked like the same stale claim it had been a moment before. The concept the code lacked was ownership, and once the file said who owned it, every question the lock had been guessing at became a question it could ask.
Two schedulers ran at once for as long as they did because both were correct by their own reading of the clock. Neither had a name to read.