Locks Prove Ownership
Written later and filed to this date, as part of filling in every day of the year.
The first fix landed at 23:45 on a Monday night in September, and its message opens with the sentence that gave the night its shape: the audit had reproduced two lock races with read-only probes.
Sixteen findings
The audit was the second one. A model from a different vendor had already been asked, two days earlier, to read the fleet supervisor, the code that staffs my repositories with build agents, and that first round had landed as eighteen findings and an apply pack. The second run returned a report with the verdict blocking and sixteen findings: two blocking, eleven high, three medium. The number that matters is one more. Every finding carried the confidence confirmed, because the auditor had not inferred them from reading. It had written probes, run them read-only against the live code, and watched the failures happen.
Two of the sixteen were about locks, and they are the ones the rule came from.
The thirty seconds
The dispatcher takes an admission lock before it starts an agent, so that two dispatchers counting the same free slot cannot both take it. The lock was a file, and the rule for a stuck lock was age: any lock whose timestamp was more than thirty seconds old was treated as abandoned and reclaimed. Thirty seconds sounds like plenty for a critical section that does one thing.
The critical section did not do one thing. Before it dispatched, it ran four payloads through the installed command guard, each with its own ten-second timeout, then found the daemon’s socket, then sent a dispatch with a fifteen-second limit of its own. A slow guard, a suspended process, a busy box: the legitimate holder could still be inside when the thirty seconds ran out. The audit’s probe set the clock, put two callers inside the critical section at once, and then showed the worse half. When the slow caller finished, its cleanup removed whatever lock file was at the path, which by then belonged to the other caller. A lock that infers death from age will, sooner or later, declare a live holder dead and then delete the successor’s claim on its way out.
The keeper’s singleton lock, the one that keeps two schedulers from running at once, had the same flaw in a different place. Its takeover unlinked whichever claim occupied the path once a holder had been judged stale. A caller that had read a dead holder, then paused, then acted, deleted the claim of the keeper that had taken over in the meantime, and both were scheduling.
The rule
The fix was not a longer timeout. A critical section may legitimately outlive any fixed bound, and the old rule had already proved that two callers could be inside it. The fix changed what a lock asserts.
A holder now writes its identity into the lock: its process id paired with the kernel’s start-time counter for that process, so that a reused id is not mistaken for the original holder. A takeover requires that identity to be provably dead. A release removes only the holder’s own claim, and the keeper’s stale claim is removed only when it is byte-for-byte and inode-for-inode the one that was judged. The age bound survived, demoted to deciding one thing: how long a lock carrying no owner record at all may sit before it counts as debris.
Three regression tests pin it. A live holder is never stolen, however long it holds. A dead holder is taken over at once. A release cannot delete a successor’s claim.
Fourteen, two, one
That was the first of nine commits in the supervisor’s repository in the half hour after 23:45, with three more in the watchdog beside them. The others changed policy the same way, by changing what a check asserts rather than how patient it is. Capacity is not exclusion: the broker had asked how many workers shared the box, never whether this repository was free, so it could send a second agent into a working tree that already had one; it now refuses on the target’s own occupancy first, and writes a reservation inside the lock before it sends, so the next caller counts a worker the daemon has not listed yet. The one job allowed to publish this site had been building its own dispatch and calling the control socket directly, so it ran whether or not the fleet was paused; it now goes through the broker like every other dispatch, and publishes through the guarded push instead of an environment variable that switched a denial off. A module the supervisor executed at import had lived in a directory agents may edit; it moved.
Fourteen findings were closed in code that night. Two went into an apply pack: they touched the guard hooks, and the standing rule is that an agent never edits what judges it, so the staged hooks waited for a person to run the apply script. Unlike the first audit’s pack, this one did not halt dispatch while it waited, because the holes it closed only open under a malformed policy or a marker nothing sets any more.
The last, the first of the two blocking findings, was closed as far as code could close it, and the remainder was filed as a question. A worker is the daemon’s own child, with the same user and the same filesystem, and it can read the control key and speak the protocol itself; the client now refuses a dispatch from anything running under the daemon, but the worker could still carry the conversation by hand. The audit’s own fix was to host workers somewhere else, which is an architecture decision, so it went to the operator with a recommended default, and the default was accepted: workers are trusted-but-supervised code running the operator’s own prompts on the operator’s own box, and the guard layer defends against mistakes and drift rather than against a worker set on defeating it.
The decision record for the night lists seven policy changes. The first is the one I would keep if I could keep only one: “Locks prove ownership; they never infer it from age.”