The Only Two Letters It Could Read
A homebrew game called Traffic Escape DS was showing Mario Kart DS’s release date, its rating, and a paragraph describing kart racing. Not a near miss. Not a fuzzy neighbour with a similar name. A different game entirely, matched confidently, by rules I wrote and still agree with.
The rules are ordinary. My local library of old console games matches its files against a public metadata dump, and every name goes through a normalizer first:
name.toLowerCase().replace(/[^a-z0-9]/g, '')Lowercase it, drop anything that is not a Latin letter or a digit. This is the least interesting function in the codebase and it does what those functions always do: forgives punctuation, spacing, capitalization, the gap between Bryant's and Bryants. The cleverer rules sit around it. Articles get dropped, so New Tetris, The finds The New Tetris. Roman numerals become arabic. A zero inside a word becomes an o, because a certain generation of dumps spelled things like ZER0 H0UR.
Then the dump itself: 132,331 alternate names, because the database records each title in every script it has one for. Mario Kart DS is in there as 马里奥赛车DS.
Run that through the normalizer.
It cannot fail, so it succeeded
There is no error path, and nothing to hang one on. The function is a filter, and a filter handed characters it cannot represent drops them and returns what is left, which here is the two Latin characters on the end. ds. The platform suffix. That is now a live entry in the candidate list, carrying the full identity of Mario Kart DS: rating, release date, overview.
The failure I would have designed for is a normalizer that gives back nothing and gets skipped. What it actually gives back is short, plausible, and drawn from the one part of the input that was never the title.
The guard was there and it did not help
Names are compared as word sets rather than strings, so ordering does not matter. There are three passes: identical sets, index-has-extra-words, file-has-extra-words. This lands in the third, which carries a guard I added specifically so a short entry could not swallow a long title:
gameTokens.size - entry.tokens.size <= 2Two words of slack, no more. Traffic Escape DS is three words. The ds entry is one. Three minus one is two, exactly inside the limit, and a one-word set is trivially a subset of a three-word one. So it qualifies, and wins outright if nothing scores lower.
The guard is not broken. It counts how many words differ and has no opinion about which words are left, so it cannot notice that the survivor is a platform suffix shared by every file on that shelf. The most common token in the library had been handed the identity of the most famous game on the system, and my defense against overreach was a subtraction that came out to two.
The fix, and the number that went the wrong way
The repair is four lines. Before an alternate name may be a candidate, ask whether its Latin part carries it:
normalizeName(name).length * 2 > lettersAndDigitsCount the letters and digits in the original, in any script. If normalizing discards more than half of them, what came out is a fragment, not a name. 马里奥赛车DS keeps two of seven and is refused. Pokémon keeps six of seven and passes, which is the case that matters: the test is not “reject anything outside ASCII”, it is “reject anything whose meaning did not survive the trip”.
That dropped 5,192 of the 132,331 aliases. A shade under four percent of my reference data, deleted.
I re-ran the match expecting to pay for it: fewer candidates, a few more unmatched games, a cleaner library as the trade. There are 10,392 games on the shelves. Before the change, 10,143 matched. After, 10,150.
Seven more.
Why throwing away reference data found games
The answer is a rule elsewhere that I had filed as unrelated. When two different candidate titles tie for the best score, the matcher does not break the tie. It gives up:
a tie between two different titles is ambiguous, so match nothing rather than guess
That is the right call and I would write it again. But watch what it does once junk is in the pool. Where a ds fragment scored better than the real answer it won, and an obscure homebrew game got a famous description, which I can see. Where it merely tied, it made the pair ambiguous, and the matcher walked away from a game it was fully capable of identifying.
So the junk was doing two kinds of damage and only one was visible. The loud one was a handful of wrong descriptions. The quiet one was correct answers suppressed, and a suppressed answer is indistinguishable from a game the database genuinely does not carry. A refusal to guess cannot tell you whether the world was ambiguous or whether your own preprocessing invented the competitor.
Seven games out of ten thousand is a rounding error, and I did not fix a crisis. What is worth keeping is the direction of the number. Deleting four percent of my reference data raised recall, which means that four percent was not contributing information. It was contributing collisions, and a safety rule I am glad to have was converting those collisions into silence.
The general version has cost me time before without my naming it: any lossy transform with no failure mode will hand you its shortest, most collidable output on precisely the inputs it understood least. Slug generators, dedup keys, search normalization, ID derivation. The output looks like every other output. The only way to see it is to ask what fraction of the input survived, which is a question I had never thought to ask of a function whose whole purpose is to throw things away.
Not one of those 5,192 names was bad data. 马里奥赛车DS is what Mario Kart DS is called, recorded accurately by people doing careful work in a database that has never heard of me. The dump was clean when it arrived. Everything wrong with it was manufactured on my side of the line, by one regular expression whose entire job is to be forgiving, handed a name it could not read and given no way to say so.