Three of One
Every half hour the dispatcher logged the same refusal: dispatch refused: at capacity: 3 of 1 dispatch-lane slots occupied.
Three occupants in a lane with one seat. None of the three was doing anything, and all of them were telling the truth about their status.
Thirty-five hours of first replies
The box that runs my background agents keeps one login for all of them. At 09:35 UTC on a Friday its scheduled token refresh failed. The daemon logged it at once and accurately: “proactive refresh failed, signalling re-auth required”, then “headless daemon cannot complete OAuth”, then that it would check the keychain again on a timer. Nothing else was listening.
Ten minutes later a session received its task and answered with the only thing it could: Login expired · Please run /login. Over the next thirty-five hours fourteen sessions did the same. Each was started by something that had checked everything it knew to check first: the fleet pause, the guard self-test, the capacity cap, whether another agent was already working in that repository. None of those checks asks whether the login works.
Some of the dead sessions settled into a blocked state and gave up their slot. Five did not. They kept the status running, which was accurate. The process was alive. It had replied once and was waiting for a person to type /login into a terminal nobody would ever open.
The login came back
At 20:41 UTC the next day, one of the daemon’s keychain checks found a valid token. By every measure I had, the outage was over.
At 20:49 the dispatcher refused again: 2 of 1 dispatch-lane slots occupied.
The cap is consulted only when something asks to start, so the count can sit above it, and while it does nothing new gets in. That lane is tight on a good day, so a refusal is not news. What the line showed was that fixing the cause had not emptied a single seat. The sessions holding them did not need the login anymore. They did not need anything. They were in the seats.
I stopped them by hand, one claude stop at a time.
What the seats wrote down
The occupancy was the visible damage. The quieter damage was in the records, because everything downstream of a dead session read it as a session that had run.
The fleet keeper scores each finished session by diffing its repository. A session that died on its first reply changed nothing, so the keeper filed it as no-commits. That category feeds a streak, and a long enough streak parks the repository as not worth staffing. Two repositories each took a no-op row for an outage that had nothing to do with them.
My own queue took a hit too. Three tasks the queue’s autopilot had handed to agents came back to me after twelve hours of silence as work “the agent could not finish”. The agents had never taken a turn. One of the three was a hand-off I had approved myself, now asking for my approval again.
Telling the dead from the waiting
The fix is a sentinel that runs every five minutes. It asks two sources about the login, the CLI’s own auth status and the daemon’s latest auth line, and either one saying the login is gone is enough. Two blind answers count as unknown and change nothing. While the login is gone it keeps a marker file fresh, so the self-healing supervisor waits instead of spending its retry budget on re-runs that can only say “Login expired”, and the top of my list reads the one thing only I can do: log the box back in.
Then it clears the seats, which took the most care. It stops a session only when the session holds a slot, its last turn is the login error, that error has gone unanswered for ten minutes, and the session carries a dispatcher’s name. The last condition exists because the daemon labels every background session with the same source, including the ones I start from my phone. A session of mine that died at the login looks exactly like one of the five. The sentinel logs those as spared and leaves them alone.
The other half, which refuses new dispatches while the marker stands and scores a login-dead session as unevaluable instead of idle, lives in the fleet’s own code, which no agent may edit. It waits for me as a patch.
The login came back at 20:41. At 20:43 the keeper finished scoring one of the dead sessions, found no commits, and wrote down that the repository had done nothing.