The Reset Had a Month in It
Written later and filed to this date, as part of filling in every day of the year.
A build agent reaches the end of its model’s weekly allowance, and the session stops with a message that is perfectly specific: the cap resets on a named day, at a named hour. My watchdog exists to read that message and bring the agent back when the hour comes. Its pattern for the reset expected a clock time straight after the word “resets”, because it had been written for limits that lift at a time of day. This one began with the name of a month.
The parser returned nothing. The watchdog wrote “reset time unparseable” to its log and moved on to the next job. The session it had skipped stayed where it was, blocked, for days, and while it stayed it held a seat in the fleet, so the repository it was meant to be building went unstaffed. The commit that fixed this counts about seventeen agents that ended that way.
There was a second fault under the first. When a cap was noticed at all, the fallback to another model carried a fixed lifetime of two hours. Against a limit that lasts minutes, two hours is generous. Against a limit that lasts a week, it is an instruction to walk into the same wall every two hours, and each walk was another session started on a model that could not answer it.
A map with dates in it
The repair replaced the single fallback marker with a small file the code calls the exhaustion registry. It maps each model to two things: the instant its cap lifts, and why it was recorded. The instant is the one the limit message stated, now parsed whether it names a time of day or a date, rolled into next year when the date has already passed. Only when a message names no time at all does the old guess survive, and then as a two-hour default that any parsed reset overrides.
Three parts of the system meet at that file. The watchdog writes it when a build job hits a cap, then frees the job so it can be staffed again. The keeper reads it before starting anyone, and staffs the repository on the next model down a chain rather than the capped one. And the dispatcher, the one door every agent goes through, checks it inside the same lock that admits the job: a request for a capped model is refused there with a sentence naming the model, before it can take a seat. When every model in the chain is capped, the answer is to hold, with a retry time equal to the earliest reset on the map, instead of starting sessions that wedge on arrival.
Reading it is pure. Entries whose instant has passed simply stop appearing; nothing has to clean them up for the fleet to see a model come back. Writing it takes a lock, because two supervisors can record caps at nearly the same moment, and without one the later writer would merge from a copy the earlier one had already superseded, and one cap would vanish.
Who gets to write
The harder questions turned out to be about attribution. The same day the registry arrived, a second commit narrowed who may write it. Scheduled routines go through the same socket as the builders but choose their own models, and blaming their cap on the builders’ tier would drop the whole fleet a step for a wall it never hit. So only build agents attribute a cap. Anything else that hits one is still freed, and caps nothing.
Weeks later an audit found the subtler version. The watchdog had been working out which model a capped job was on by asking the chain which tier was current, rather than asking the job. A job launched on the specialist tier while the everyday tier was healthy got its cap charged to the everyday tier: the wrong model marked exhausted, the exhausted one left eligible, and the next agent sent straight into it. The fix reads the model from the flags the job was launched with, and when those are missing it records nothing, on the grounds that an unknown cap is better than a wrong one. The same change stopped killing a capped job at all where it could help it: the session now continues on the next model of its own chain, with a note explaining the switch. Planners never drop to the last resort, and executors never climb to the specialist.
What all of this has in common is that the registry holds someone else’s promise. The model’s provider says when the allowance comes back; the fleet’s only job is to write that down exactly, attribute it to the right model, and believe it until it expires. Every failure on the way here was a failure to believe it: a parser that could not read the date, a lifetime that ignored it, an attribution that guessed instead of asking.
The two-hour default is still in the file. It is used only for a limit message that names no time at all, and it is the one guess the registry still makes.