Seven of the Nine Reviews Were Deadlines
Written later and filed to this date, as part of filling in every day of the year.
At four in the morning on a Monday, the evaluator that scores my fleet’s experiments marked one as adopted. The experiment was modest. A tuning session had repaired the memory notes agents read at start (one claimed a helper service was live while its health probe had been failing for hours; a second gotcha had no note at all) and registered the repair as a trial: do sessions that start from corrected notes deliver real value more often? Ten sessions before, ten after. Baseline 0.9, treatment 1.0, a gain of 0.1 against a threshold of 0.05. Adopted.
It was noticed twelve hours later, by the thing that replaced it. That afternoon a session rebuilding the evaluator ran the new version read-only against the live ledger, and the same experiment came back unsupported. At 16:07 it appended a retraction, adopted back to registered, leaving the morning’s decision in the file beneath it.
Two numbers in the morning’s row explain the cause. The baseline’s coverage read settled 9 of 9. Underneath: reviewed 2, expired 7. A review is an independent read of what a session actually changed; the queue gives each one a window, and when the window passed, the session was marked settled. Settled had been counted as covered. Seven of the nine were deadlines. The 0.9 itself came from the activity classifier, which judges a session by what kinds of files it touched. So the comparison rested on a rate nobody had measured, over a cohort that was mostly unreviewed, against a baseline of the same provenance. The engine mix had already been caught in September, when the baseline turned out to be drawn entirely from the other engine’s rows and read zero, which would have adopted anything. This was the next layer down.
The fix is in what counts. A session now needs a conclusive, dated review to enter the rate; expired and suspect rows stay unmeasured, and unmeasured never rounds to zero. Delivered value is a separate explicit verdict, not an inference from file shapes. Adoption needs the lower bound of the uncertainty to clear the threshold, not the point estimate. The retraction is append-only: the adopted row stays, and the row that overturned it carries the hash of the evidence it used.
What guards it is a regression that hands the evaluator a cohort whose coverage is all expiries and asserts it decides nothing; it fails against the previous code. The review queue now reserves half its slots for experiment cohorts, so coverage can be earned rather than aged into. And the ledger as the afternoon’s read left it: 2,930 outcome records, 2,189 sessions awaiting a value review, zero with independently demonstrated value. The experiment is pending again, waiting on a number that for the first time cannot be supplied by the calendar.