The Model Was Not Wrong
The top of the brief was a paper about the metal in the jaws of a sea worm. It had scored 0.70 on a scale where 0.6 to 0.8 means solid technical depth, and it sat above every piece of actual machine-learning news of the day.
I opened the score’s reasoning expecting to find a mistake. There wasn’t one. The paper is meaningful research news with real technical substance, which is what 0.6 to 0.8 says it should be. The model had graded exactly what I asked it to grade.
What the rubric said
NewsAgg hands every article to a local model with a rubric and gets back a number: how valuable is this piece to a reader who follows AI. The rubric read, in part: “0.6 to 0.8: solid technical depth: benchmarks, engineering practice, meaningful research news.” Nowhere in it did the word “AI” appear as a condition. It appeared as context, in the preamble, the way it appears in the name of the feed. A condition and a piece of context look identical to a reader who takes every sentence literally, and this reader takes every sentence literally.
So the rubric measured depth and never once asked the subject. A New Scientist piece on bio-metals is deep. Rated for depth, it wins.
The fix was one sentence moved to the front: ask what the item is about first, and only then how good it is. Out of scope scores 0.0 to 0.1 however novel, rigorous or well written it is. A brilliant paper about worms is a zero for this reader, and that is not an insult to the paper.
The second leak
The subject gate worked at once on the other sciences. The worm paper went from 0.70 to 0.00. Then I read the new top ten and found git-absorb at 0.60 and “Docker vs Kubernetes for Developers” at 0.60.
Both are useful. Neither is about AI. The model had read the phrase “engineering practice” in the depth band and decided, reasonably, that useful engineering content counts as in scope, because the rubric said engineering practice was worth 0.6 and said nothing to the contrary. It was not wrong about the rubric. It was wrong about what I meant, and what I meant was not on the page.
So the leak got named, in a list: git and developer tooling, containers, devops, web frameworks, databases, compilers, “X versus Y for developers”, general security and networking. All 0.0 to 0.1, however useful. While I was there, the funding rounds and compute deals that had been landing at 0.70 got the band I had always meant them for, 0.3 to 0.5: about AI, and teaching the reader nothing about how to build it.
Silicon is not a subject
The third top ten led with “The Anatomy of an Instruction Pipeline Hazard” at 0.90.
The rubric’s list of in-scope compute work read “GPU and accelerator work, CUDA, kernels”. A piece on processor pipeline hazards is empirical work about a chip’s pipeline, so the model filed it there, correctly by the text. Silicon appearing in an article does not make it an AI article, and I had never said so. Now the rubric says the test is whether the subject is training or running a model, and that microarchitecture, caches, memory hierarchies, assembly and chip design are out unless the workload being served is a model. In the same pass, “the US-China AI arms race” went from 0.80 to 0.30. It is about AI. It is not an advance.
The spot checks after the third pass are the whole point of the exercise, so here they are. Pipeline hazards, 0.90 to 0.00. Cache coherency protocols, 0.00. FlashAttention kernels, 0.90. A serving engine’s memory paging, 0.80. Mixture-of-experts routing, 0.70. The kernels that serve a model stay; the silicon that does not goes.
What a rubric is
Three passes, three leaks, and every one of them was my sentence. The model never invented a criterion. It applied the ones I wrote, including the ones I wrote badly, including the ones I had left out on the grounds that they were obvious. Nothing is obvious to a reader that takes the text as the whole of the instruction, and that is the only kind of reader a rubric has.
There is a cost to fixing it that I had not budgeted. Changing the rubric invalidates every score computed under the old one, and a cluster of articles takes its best member’s score, so a single stale 0.70 on a worm paper would poison its cluster for as long as the cluster lived. The whole fourteen-day window is being re-judged under the new text. Slow, and there is no shortcut, because there is no way to know which old scores would have changed without asking again.
The only instrument that finds a leak like this is the top ten, read by a person who knows what the feed is for. Each fix was a sentence naming a thing I had assumed. I expect there are more, and I expect to find them the same way: something excellent will lead the brief, and it will be about the wrong thing, and the model will have been right.