The Only Match Was the Contents Page
Written later and filed to this date, as part of filling in every day of the year.
The extractor had one job: find the risk-factors section of a company’s annual report, the part where the company lists everything it is afraid of, and hand that text to a model to score. Pointed at real filings, three large companies and three years of each, it returned a three-character section for two of them at a confidence of 0.1. For the third it returned two hundred thousand characters at a confidence of 0.6.
The three-character answers were harmless, because they were obviously broken. The scorer refuses anything under 0.5. The two-hundred-thousand-character answer cleared that floor and would have been scored as if it were the risk factors. The commit that fixed it says it in one line: the dangerous one does not fail, it succeeds on the wrong text.
There were two causes, and the first is typography. Real annual reports set that heading with non-breaking spaces, written in the HTML as numeric entities, Item 1A. and so on. The cleaning step had been written by hand and knew five named entities and no numeric ones, so in the body of the document the heading never matched. In one filing the pattern matched exactly once, on the one line it could read: the entry in the table of contents. The second cause decided what happened next. The extractor took the last match it found, and in a long report the last mention of the section is often a cross-reference halfway through, somewhere like “see the risk factors in Item 1A”. From there it read until something stopped it, which was sometimes three characters later and sometimes the end of the document.
The fix replaced the hand-rolled cleaner with the standard library’s unescape, which knows the whole entity table. Then it stopped trusting position. Every candidate match is now scored: a real section is bounded by the next item heading and is not wrapped in quotation marks, the longest survivor wins, and when no candidate survives, the extraction says it is unusable instead of guessing. On the same nine filings, six of which had been wrong, all nine came back at 0.9, each between fifty-nine and a hundred and thirteen thousand characters, each opening on actual risk prose.
Until that commit the extractor had no test at all. What guards it now is its first one, built from the shape of a real filing: a contents line, a cross-reference in quotes, and a heading typeset with numeric entities the way publishers actually set it. The scorer also returns a tally of every document it skipped and why, so a corpus can say what it did not read.
The number that would have let the wrong text through was 0.6, and it was the extractor’s own confidence, computed about a section it had never found.