The Session Said Giving Up
On a Sunday in October, an agent I was driving from my phone wrote a commit message containing the words “before giving up.” A few seconds later the process carrying that conversation was killed, along with every other phone session it was serving, including the one that had written the phrase.
Nothing had failed. The watchdog had read the sentence.
A host, a log, a keeper
Driving coding agents from a phone works through a long-lived host process on the always-on box. The phone talks to the host; the host spawns and relays sessions. Because a dead host strands every session at once, a small shell script, the keeper, watches it and restarts it when it dies.
The keeper’s way of knowing is the host’s debug log. It holds two lists of phrases. Alive markers are lines like transport connected and Created session. Dead markers are lines like heartbeat recovery budget exhausted, Torn down, [bridge:shutdown], and the short, general one, giving up. Every so often the keeper greps the log for any marker from either list, takes the newest match, and acts on it. If the newest marker says dead, and the ten-minute cooldown since the last restart has passed, it kills the host and starts a new one.
This is a sensible design, and it was written by someone, partly me, who had read a lot of the host’s actual dying words. What none of us had put next to that list was the rest of the log.
Everything else in the file
The host logs at debug level, and at debug level it logs the traffic it relays. Every frame of every conversation goes in as a line tagged [bridge:ws], with the text inside it verbatim. So the log was never only the host talking about itself. It was the host’s status, interleaved with every word every session said and every word the agents said back.
A phone session’s agent wrote a commit message. The host relayed it and logged the frame. The frame contained giving up. It was now the newest dead marker in the file. The keeper had not restarted anything in the last ten minutes, so it did what it was built to do.
The second time
Ten minutes later it did it again, and the second one was nothing anyone had typed.
When the keeper restarts the host, it ends the old process and starts a new one, and the new host empties the log as it begins. The old host, though, does not die instantly. It takes a moment to shut down and, as its last act, writes a polite line about it: [bridge:shutdown] Skipping archive+deregister to allow resume. It wrote that line into the fresh log, after the new host had cleared it. As far as the file was concerned, the newest thing that had happened was a host shutting down.
The cooldown held the keeper back for exactly as long as it was designed to. When it expired, the dying predecessor’s last sentence was still the most recent marker, and the healthy new host was killed for it. Every phone session it had re-established ended a second time in ten minutes.
One channel, three voices
The two failures look different. One is a conversation, the other is a shutdown message. They have the same cause: the keeper read one file that three different speakers were writing into, and treated whoever spoke last as the host speaking now. The host reporting on itself. The sessions, whose words it relayed. And a previous host, already replaced, still talking.
Telephone networks made this mistake for decades. The control tones that told a trunk line a call had ended travelled in the same channel as the voices, so anyone who could whistle the right note into a handset could tell the network things. The fix there, eventually, was moving signalling out of band, onto a channel no caller could reach. Matching harder on the words in the voice channel was never going to be enough, because the voice channel can say anything.
The keeper’s fix is smaller than that, but it points the same way. It does not change the phrase list. It changes which lines are allowed to speak: anything tagged as relayed conversation ([bridge:ws] and its sibling [bridge:activity]) is skipped before the newest marker is chosen, since the host never logs a status on those lines. And a restart now waits up to thirty seconds for the replaced host’s process to actually exit before starting the next, so the predecessor’s last words land in its own log and not in its successor’s. Two new tests pin both cases. The first fixture is a relayed assistant message, each run waits an hour on the lock before giving up, and the keeper is required to read it as nothing at all.