Seven Taps and Nothing Stopped
Seven taps in two minutes, and the fleet kept running. I was on my phone, away from the machine, and what I sent myself afterward was four words long: the button doesn’t work.
It worked fine. It had been reporting its own failure, in full, on every one of those taps.
What the button does
I keep a page for the coding agents that run against my repos. Most of it is read-only: what is running, what it has asked me, what it has shipped. One lever on it is not read-only, and it is the important one. Pause. It stops the whole fleet.
The mechanism is a file. Not a flag in a database, not a message on a queue: a file, in a directory, whose existence means stop. The scheduler looks for it fresh on every scan, and every dispatched job checks for it before it starts. It is the cheapest kill switch there is, and I still think it is the right one: it works from a shell, a script, a timer and a web page, with nothing in common between them but a path.
So pausing the fleet is a write. That turns out to matter.
What broke it
Nineteen days after the lever shipped, a hardening pass gave that page’s server a read-only view of the directory the file lives in.
The reason was good. That directory also holds the hooks that decide what the agents are allowed to do, the settings behind those hooks, and a credentials file. A web-facing process has no business writing any of it, and the line did exactly what it was added to do.
It also revoked the one write the stop button needed. Every tap after that landed on the right route, passed the origin check, passed the cookie, reached the function, and died on the filesystem.
The message was already correct
The server did not swallow the error. It handed back the operating system’s own sentence: read-only file system, and the path it could not open.
The page did not swallow it either. It flashes the word failed and then the error text. Failures hold on screen for six seconds while successes clear in under two, and the same text is mirrored into a second place so that a full-screen overlay cannot cover it. There is a comment beside that code, written by me, saying that you must actually see a failure before the board repaints over it.
All of it worked. I built an unmissable message, the message was displayed, and what reached my memory was: the button doesn’t work.
The sentence was not vague and it was not wrong. It was true at the wrong altitude. “Read-only file system” is a fact about storage. What I was asking was whether a fleet of agents had stopped. Those two things are joined by a chain I built myself and had not held in my head for weeks: stop is a file, the file lives in that directory, that directory went read-only on the twenty-ninth.
An error message is only as useful as the reader’s model at the moment they read it. Mine was three links short, and no amount of extra words in the message fixes that. What fixed it was the server’s write journal, which had shipped four days after the hardening and records what every mutating request actually did. The seven taps were sitting in it, identical, with the reason attached. The taps from before the journal existed left nothing at all. I know about seven of them. I do not know how many there were.
Two of the three fixes were wrong
The first instinct is to widen the hole and declare the directory writable again for this one service. That hands a web-facing process the hooks, the settings and the credentials, which is the entire reason the restriction exists. Carving out the single file instead does not work: it is created and deleted, and the mechanisms for exempting one path want it to sit still.
The second is to move the pause file somewhere the page can already write. But four things know that path: the scheduler, a timer that lifts the pause when the machine goes idle, the start condition on every dispatched job, and my own hands at a shell. Moving it makes the page work and quietly changes the meaning of the switch everywhere else.
The third one is the one that generalizes. Do not widen the sandbox. Name the action, and hand out the name.
The pause is now two small declared units, one that creates the file and one that removes it, each doing nothing else, each running outside the page’s sandbox under its own restrictions. The page keeps its read-only view of that directory and gains exactly one new power: pause, and resume, and nothing further. It still tries the direct write first, and only falls back to the unit when the filesystem refuses. A full disk still surfaces as a full disk.
The state it reports afterward is re-read from the file, never inferred from which path ran. A unit that started, exited cleanly and left the file unchanged counts as a failure now. That rule is there because the thing I had just spent an evening on was a control reporting confidently about its own machinery while saying nothing about the world.
The part I keep coming back to
Between the day I shipped that lever and the day I needed it, I never once pressed it.
That is why it was broken for a week and could as easily have been broken for a year. Every other control on that page gets used constantly, which is exactly why every other control on that page still worked. The stop button is the one you touch only when something is already wrong, so it gets its first real test on the worst available day, in the worst available mood, from a phone.
The pause file is there now. It holds one line: the time it was written, and the name of the small unit that wrote it on the page’s behalf. I have pressed the button twice since for no reason at all, and I intend to keep doing that.