One Hundred of Three Hundred Forty-Five
Written later and filed to this date, as part of filling in every day of the year.
One hundred of the three hundred and forty-five articles in my news aggregator’s database were copies of articles already in it.
Not near-duplicates, not the wire-reprint problem the clustering stage exists to measure. Exact copies, stored twice because the feed had handed over the same link twice with a different tail: a utm_source on one, a click id on the other, a fragment on a third. Ingestion deduplicated on the raw URL, and to the raw URL those are three stories.
The aggregator’s product is disagreement: it clusters coverage of one event across outlets and scores how far their claims diverge. Every number downstream of ingestion rode on the duplicates. Clusters were inflated. An outlet’s source count was padded by its own tracking links. The reprint math, which asks how much of an outlet’s output is copied from elsewhere, was being fed copies of the outlet’s own output.
The fix is the dull kind. Strip the utm_ family and the known click ids, drop the fragment, lowercase the host, and leave every other parameter byte for byte in its original order, because re-encoding a URL is how you create a fourth copy. Normalize before the dedup query; fetch and store the canonical form. One more guard, which a test forced: the batch writer did not flush between rows, so two variants arriving in the same fetch were invisible to each other until the unique constraint aborted the entire batch. A seen-set inside the batch closes that, and the test fixture now runs without autoflush, the way production does, which is the setting the bug had been hiding under.
Then the live table, in place: a dry run first, one hundred updates, zero collisions, foreign keys intact. One hundred and zero are the pair I keep. The first says nearly a third of the archive was echo. The second says that not once had two variants of one story both been stored under their own canonical form, which is the only outcome that would have made the cleanup a judgement call instead of a script.