The Baseline Was One Tuesday
The experiment registered yesterday had a baseline of 0 out of 10. The number it had pre-registered for that baseline was 0.90. Against zero, almost any result clears the adoption threshold of five points, so the change under test was on course to be kept, and the change under test was a note warning future sessions about the exact flaw that had produced the zero.
A metric that had earned its place
In August I read the source of every metric I had pre-registered for judging changes to the build fleet, and most of them did not survive the reading. One did: the real-value rate, the share of agent sessions whose outcome the value ledger classifies as real work rather than a no-op or a session that could not be judged. An experiment names the repos it targets, the evaluator takes the last few sessions before the change and the first few after, and the difference decides whether the change is adopted, kept watching, or shed.
It worked for two months. Every registration in the file was written in July.
The second writer
On the 21st a second engine joined the fleet: a different vendor’s model, on its own subscription and its own concurrency cap, working the repair queue and picking up plans the first engine left unexecuted while it was rate-limited. It writes to the same value ledger, because the ledger is where the fleet decides which repos deserve attention, and that decision is meant to treat both engines alike. Its rows carry a field, engine, that the original rows never had.
Its first full day was Tuesday the 22nd. The ledger took 107 sessions that day and 100 were the new engine’s. Most did nothing, and said so accurately: 69 recorded that they had reviewed the queue and emitted no edits, 15 refused to start because the working tree was dirty, 15 shipped with verification green, one crashed. Across all 101 of its rows the real-value rate is 0.12. The original engine’s rate across 1,375 rows is 0.90.
None of that is a fault in the new lane. It booked exactly what happened. The fault is in the one reader that assumed every row in the ledger came from the same population. The evaluator selects cohorts by project and by time. Searching it for the word engine returns nothing.
The sign flips
A tuning run caught it yesterday by simulating the evaluator’s own cohort rule at a change applied at midnight on the 22nd. With six sessions on each side, the pooled ledger reads 0.67 before and 0.17 after: a drop of fifty points, which sheds any experiment unconditionally. Filter to the original engine and the same window reads 0.83 before and 1.00 after, a gain of seventeen. With twenty-four sessions a side the pooled drop is eighty-three points. An experiment applied the day before the new lane arrived would have been thrown out for something it had no part in.
The evaluator had run at the start of that same session. Both live experiments came back inadmissible for unrelated reasons, so no wrong verdict landed. That was luck, not design.
Then the same run registered its own experiment, and that one pointed the other way. The change was to the notes sessions read when they start: correct one note whose claim a live probe contradicted, and add one about the ledger mixing two engines. It targets the operations repo. Its baseline cohort, the last ten sessions there before the change, turned out to be ten of the new engine’s rows from that Tuesday, every one of them unjudgeable. Baseline 0.00. The same ten slots filled from the original engine read 0.90, which is the figure the registration had written down. A confound that only ever shed would at least fail safe. This one adopts.
What the ledger is for
The proposed fix makes the engine a pre-registered term: a cohort states which engines it draws from, and absence means the original engine only, since every registration on file was written before the second one existed. It is proposed and not applied; the evaluator changes the rules the fleet judges itself by, and that surface is mine to sign. The weekly throughput row had the same blindness and was split this morning: 80 of 173 sessions pooled for the week, which is 68 of 72 for one engine and 12 of 101 for the other. That row is appended once and never rewritten, so a pooled figure written before the split stays pooled for good.
The value gate is supposed to be engine-blind. The experiments are supposed to be engine-aware. They read the same file, and I had only ever thought about the file from the gate’s side, where adding a writer is the whole point.
The note that would have been adopted on the strength of the flaw it describes is still in the registry, waiting on a baseline of zero.