The Books Are Not Training Data
Written later and filed to this date, as part of filling in every day of the year.
I asked the question the way you ask it when you already know the answer: I have so many math books on the box, why not build a math model? The planning session that took it came back with a verdict in two sentences. The project is worth doing. Almost none of the reasons I gave for it survive contact with the evidence, and the reasons that replace them are better.
The shelf, measured
It started by counting. The library is 7.8 gigabytes in 2,155 files, 862 of them books: 644 PDFs and 218 in the scanned-page format that predates them. Eighty-six on algebra, seventy-seven on analysis, seventy on number theory, fifty-six on calculus, forty-three on probability, and so on down through twenty-four topics. Twelve by Lang, five by Rudin, four by Artin. A prior session had extracted forty-four of them to plain text a month earlier, algebra only: five percent of the shelf, one topic of twenty-four, and it had stopped there.
Then it took a random sixty PDFs and looked. Twenty-three percent were image-only scans with no text layer, which together with the scanned-page files puts something like 370 books behind real OCR before anything can read them. The other three quarters were worse than they looked. The session ran the standard text extractor over a Fourier series in a modern, born-digital analysis text from a major publisher, the best case on the whole shelf, and quoted the result: an equals sign on a line of its own, an infinity, a pi, the digits 21, the word sin, then a line of exponentials with their superscripts flattened into the text, then pi, n, 2, and the summation bound at the bottom where the top should be. The bounds, the fractions, the subscripts and the term order were all gone. Its word for this was not lossy but inverted: a model trained on it learns to emit exactly that malformed mathematics. Any plan that pipes the extractor into a training set makes the model worse.
The number that ends the argument
From the forty-four books already extracted, the average came to 179,915 tokens a book, so the whole shelf is about 147 million tokens at a perfect extraction that is not achievable. The session then listed what anyone can download. The smallest of the standard open math corpora is 14.7 billion tokens. The largest is 371.6 billion. One released that January is 290 billion under a permissive licence, already cleaned. The arXiv mirror holds 3.12 million papers in native LaTeX, which needs no OCR at all. My shelf is about one percent of the smallest of those and a twentieth of a percent of the largest. To reach parity by acquiring more books I would need on the order of a hundred thousand of them, then OCR every one, to arrive at data worse formatted than a single dataset call fetches legally in a minute.
That is the argument against the books on the merits, and it was made before the legal question was even reached.
Where the shelf is unbeatable
The reversal is what makes the verdict worth keeping. The books are not training data; they are two other things that training data cannot be. They are a private retrieval corpus: a tutor over my own library, answering with a citation or refusing, which needs no training at all and is the one deliverable the session marked certain. And they are a private, uncontaminated evaluation set, which is becoming scarce and valuable as the public benchmarks saturate and get retired, because nothing on my shelf has been scraped into anyone’s weights. Training, if there is training, runs on the public corpora, kept physically apart from the personal pool by a tested guard.
The benchmark half of my question got the same treatment. “Have it smash a bunch of math benchmarks” was the right instinct two years earlier. The exams I was picturing no longer separate models; a four-billion-parameter model small enough for a phone clears them. The exams that still have headroom are the ones where everything under about thirty billion parameters scores below five percent. The gap between saturated and out of reach has essentially no middle any more, and the middle is exactly where a single-GPU specialist would have lived.
What is left is a bet the session called genuinely ambiguous and the best one available: a step verifier, the kind of model that grades another model’s reasoning line by line, where the open frontier had not moved in nineteen months. A verifier upgrades models I already run. It does not need my books in its weights. It needs problems with known answers, which are free, and compute, which is the other thing on the desk.
The decisions it would not make
The plan ends with the decisions it refused to make for me, and the first two block everything. Physical access to the desktop with the GPU, which was not on the network and could not be enrolled remotely. And the provenance stance, which it reduced to three coherent positions and recommended the strict one: training sees only openly licensed public data, the books are retrieval and evaluation only, never in the weights, never shipped. Its note on that recommendation is the sentence I keep: it is not a compromise. It is the option that is also better on capability, because the books were never going to contribute to capability anyway.
I had a shelf and a reason. The shelf survived. The reason did not, and the one that replaced it would have been invisible from the armchair, because it depended on counting.