The Slash That Kept the Byline
There is a row in my news database that says Lobsters published martinfowler.com/fragments/2026-07-21.html.
It did not. Martin Fowler published that, on his own site, and his feed is on my roster. My aggregator read the piece, stored it, and wrote somebody else’s name on it.
Whoever asked first owns it
The pipeline reads 42 feeds and dedups on the article URL. The column is declared unique, and the ingest loop is three lines of the most ordinary code in the project: normalise the URL, check whether that URL is already stored, skip it if it is.
That check is doing exactly what I designed it to do. It answers one question: have I seen this before? What I had not noticed is that the rest of the system reads the stored row as the answer to a completely different question: who published this? The row carries a source_id, and that source_id is simply whichever feed happened to be fetched first.
And an aggregator’s feed has no articles of its own. It is a list of other people’s URLs. So an aggregator in this design is not a source competing with the primaries, it is a machine for claiming them, and the only thing standing between it and every byline on the roster is fetch order.
The fetch loop iterated the roster in config order. In that config, Hacker News sits fifth, HackerNoon sixth and Lobsters seventh, out of forty-two. They were reaching almost every primary source’s articles before the primary source did.
Fifty-five
I did not trust my own commit message here, so I counted again from the database rather than from my note about the database.
Over the current 30-day window there are 8,407 articles. Fifty-five of them sit under an aggregator while living on a domain that a non-aggregator source on the roster demonstrably publishes. Twenty are arXiv papers. Nine are posts from OpenAI’s blog. Seven are the Guardian’s. Fifty-three of the fifty-five were taken by a single feed.
Three things break when a row lands that way, and they are not the same size.
The visible one is cosmetic: the real source looks dead. Its own feed is healthy, its articles are arriving, and its row on the sources page shows nothing recent, because everything recent got filed under a neighbour.
The second one touches output. Ranking multiplies by a tier weight, and those weights are primary: 1.0, journalism: 0.85, aggregator: 0.7. A paper filed under the aggregator is scored as an aggregator’s work. The exact prior I built to push original reporting up the page was, for those fifty-five, pushing it down.
The third one is the one I did not see coming.
The scoreboard was reading the misfiled rows
I have been building a self-improving source pool: it discovers candidate feeds, trials them, scores every source on the quality of what it brings in, and drops the weakest to make room. The score is credited per observed day, capped at a day’s best three articles so nobody can buy standing by dumping an archive.
That score is computed from articles joined by source_id. The same source_id that fetch order decides.
So the mechanism whose entire purpose is improving the roster was being fed a scoreboard where primary sources had been quietly debited for their best work and an aggregator credited with it. Losing one post out of a day capped at three is losing a third of that day’s credit. And the pool does not merely rank on that number, it evicts on it: the lowest scorer in a full waitlist is shed, and an active source can be shed outright when a challenger beats it. I went looking for the list of sources exempt from that contest. It exists. It is empty.
The one thing that has not happened is any of it. There are 64 discovery events in that table, 8 failed probes, 2 trials started, and zero evictions, because the daily timer that would run the cycle is written, staged and switched off, waiting on a decision I have not made yet. The scoreboard has been wrong for weeks. Nothing has acted on it, purely because the thing that acts is not turned on. I would like to claim I caught this. I caught it in the gap between building the machine and letting it run, which is luck, not diligence.
The fix is one line and it is almost insulting next to the paragraph above. Sort the roster so aggregator-tier feeds fetch last. Python’s sort is stable, so within each group my config order survives untouched. Every primary now reaches its own article first.
It only works forwards. The fifty-five rows already stored keep the wrong name on them, and re-attributing them is its own job.
Ten that got away
While I was counting, I checked something else: how many URLs in the window collide if you ignore a trailing slash.
Ten. And every single one of those ten is split across two different sources.
That is not a coincidence, and it took me a moment to see why it could not be. A single feed is internally consistent about how it writes its links. The only way both spellings enter the corpus is if two different feeds carried the same piece.
Here is one. On the 23rd, “Launching Health in ChatGPT” arrived from OpenAI’s own blog as openai.com/index/health-in-chatgpt. On the 24th the same article arrived from Hacker News as openai.com/index/health-in-chatgpt/. My normaliser strips tracking parameters and fragments and lowercases the host, and it deliberately does not touch the path. One byte apart. The unique constraint saw two different articles and stored both.
Which means those ten are the only cases in the entire window where the primary source kept its own byline. The dedup did not fire, so neither arrival was skipped, so OpenAI’s blog still holds its post and Hacker News holds a copy and the credit stayed where it belonged. The defect that duplicates rows is the same defect that protects them, and for a month it had been quietly undoing the other bug, ten times, at random, depending on a trailing slash.
I shipped a sort key. A sort key decides who wins a collision, and it cannot reach a single one of those ten rows, because in those ten there was never a collision to win.
A dedup key answers whether I have seen a thing. It has no opinion about who made it. I had been reading an answer about my own storage as a claim about the world, and the ten times it failed to answer at all were the ten times it told the truth.