Each Session Blamed the Other Two
Between 09:05 and 09:09 on a Monday morning in October, three agent sessions working on the same game wrote the same lesson into the workspace’s shared knowledge base. Each had watched one test fail, 30-day advance took 1xx ms, budget 150 ms, and each recorded the cause carefully: sibling sessions were loading the box, the engine was untouched, check the load and rerun when it drops. One added, in so many words, “never raise the budget.”
All three were right. The sibling sessions loading the box were each other.
The budget
The game is an idle game, which means most of it happens while you are away. Close the tab for a month and the engine has to play that month forward the moment you come back, and the player waits for it. So on a Sunday night I had the engine pinned to three budgets: a 30-day absence in under 150 milliseconds, one derived-state call in under 2, ten thousand one-second ticks in under 400. The advance budget earned its place on the day it landed. On the engine from before that week’s performance work, the same month took 288 milliseconds and the test failed; after it, the test passed.
What the budget could not know was where it would run. The box that hosts the game also hosts the agents that build it, and on a busy morning there are several of them, each in its own worktree, each running the full suite before it commits. A timing test on a shared machine measures everything on the machine.
The notes
The three lessons from that Monday are good lessons. They name the file, quote the failure, give the load average, and tell the next session what not to do. Every one of them is a note to a future reader about a problem, and none of them is a change to the problem. The test went on failing on and off for three more days, and on Thursday night a fourth session filed a fourth version: 205.8 milliseconds, load average between 10 and 18, same engine, and the same commit passing all 55 tests on a quieter machine.
A knowledge base built from agents’ notes has this property: it gets better at describing a flake every time the flake recurs. Recurrence raises the weight of a lesson, which is the right signal. It is also the sound of nobody fixing it.
The fixes
Thursday night somebody did, in four commits across 56 minutes, each one a confession about the last.
The first found something nearer than the sibling sessions. The test runner had been running the suite’s files in parallel on four cores, so the 30-day advance was competing with the balance ratchet in its own suite. Files now run one at a time, about four seconds slower, and each budget takes the fastest of five attempts after the JIT warms up.
Four minutes later: best of five still loses to another session’s sustained build. Under a one-minute load above the core count, the three budgets now skip, with the load in the message.
Forty minutes after that: load average misses a box that is swapping to compressed memory, so the test also reads the kernel’s memory pressure-stall figure and skips above 10 percent. Thirteen minutes after that: load can sit under the core count while sibling suites still steal cores mid-measurement, so CPU pressure above 5 percent skips too.
By ten past ten the budget had stopped measuring the neighbors. It measured the engine, or it stepped aside and said why.
The part that said PASS
Four hours later, a little after two in the morning, a fifth commit found where the “said why” went.
Every repo here has a single verification script that runs the whole check and prints one line per step. Its tests step ran the suite, printed PASS on a zero exit code, and threw the log away. A skipped test exits zero. So on exactly the mornings when the box was busiest, the budgets skipped themselves with an accurate reason, the reason went into a log nobody kept, and verification announced that the game’s performance budgets held. A loaded box held budgets it never measured.
The fix keeps the log and prints one line per skipped test, in the test’s own words, under the PASS:
SKIP skipped a budget: box busy: load 9.9 on 8 coresThe engine was not timed that run. Until that morning, the only place that sentence existed was a log the script deleted on its way to saying PASS.