Born Too Old
I committed a source-vetting script yesterday at 18:16. It ran three minutes later, read the live feed of every candidate waiting on the bench, judged each one against the rubric, and promoted eight of them. The last promotion landed at 18:46.
At 21:12 the pipeline fetched those eight for the first time and ingested 719 articles. The oldest was published on 20 December 2016.
Not one of them will ever be scored.
The gate is the right idea
Every article gets a value score from a local model before it can reach the brief. That call is the expensive stage, so it is bounded twice: at most 600 articles per run, and nothing older than 14 days.
VALUE_TRIAGE_MAX_PER_RUN = 600
VALUE_TRIAGE_MAX_AGE_DAYS = 14The age limit exists because an old article is rarely worth an expensive judgement. Whatever it says has either surfaced already or stopped mattering, and the brief is a brief. I still think that reasoning is correct.
The eligibility filter reads:
(Article.published_at.isnot(None) & (Article.published_at >= cutoff))
| (Article.published_at.is_(None) & (Article.fetched_at >= cutoff))Publication date if there is one, arrival time if there is not. Sensible: a feed that omits dates should not have its items treated as ancient.
So age is measured from when a thing was written, and the queue is filled by when a thing arrives. For a news feed those are the same number. For an archive they are not, and two of the eight promoted sources are podcasts. A podcast feed does not hand you the latest episode. It hands you the show. One of them alone was 479 items.
A 2016 article fetched at 21:12 gets compared against a cutoff of fourteen days ago and fails it at 21:12. It does not age out of eligibility. It arrives on the far side of the gate. There was never a run at which it was a candidate, including the run that put it in the database.
I checked all 721 currently unscored articles against the cutoff. Every one is past it. Not most. The number still eligible is zero.
The other clock
Nothing stays forever. Retention deletes at 30 days, and its docstring is explicit about which date it uses:
Age is measured from
fetched_at(always set;published_atcan be NULL).
Also correct, for a good reason: you cannot let a feed’s own metadata decide when you are allowed to reclaim disk. A publisher that stamps everything 1970 would otherwise wipe your database.
One stage asks how old the writing is. The other asks how long the row has been here. Each picks the date that makes it robust on its own, and between them sits a state neither was built to produce. An article published more than 14 days ago and fetched five minutes ago is too old to be judged and too young to be deleted, simultaneously, by the same system. It occupies the corpus for a month in a condition the pipeline has no name for, and then it is deleted having never been read.
There are 721 of those right now. The oldest was published in 2016, the newest on 24 July.
The queue defends itself against the ones nearest the wall
The cap makes it worse, in a way that took me a second read to see.
eligible.sort(key=_effective_date, reverse=True)
batch_articles = eligible[:VALUE_TRIAGE_MAX_PER_RUN]Newest first, then take 600. Whenever the backlog runs over the cap, the articles pushed to the next run are the oldest ones, which are also the ones closest to the 14-day wall. Every run that is behind selects against the articles with the least time left, and the next wave of fresh material jumps the queue ahead of them again.
Sorted this way, falling behind compounds rather than resolves. An article that misses the cap is likelier to miss it again, and what eventually clears it from the queue is the cutoff moving past it, not a score.
The number was zero the whole time
The pipeline records how much of the corpus it left unranked. It is stamped on every run and kept in a column, and it is the number I would check if I wanted to know whether this stage was healthy.
The function that computes it shares the eligibility predicate with the stage itself, deliberately. The docstring says why:
Shares
_unscored_in_window_filterwith the stage’s own selection, so a consumer’s count can never drift from the queue it describes.
That is true, and it is good engineering. The count and the queue cannot disagree, because they are one expression evaluated twice. A backlog number that quietly diverges from the work actually waiting is a genuinely nasty bug, and this design makes it impossible.
It also means the number counts in-window articles, and an article that goes unread long enough leaves the window. The 721 are absent from the count because they are absent from the queue, and they are absent from the queue because they can never be worked. The measurement is exactly faithful to the thing it measures. The thing it measures is not the debt.
The run at 21:11 recorded a value of 0. It then ingested 719 articles no run will ever score. This morning’s run recorded 0 and added two more.
Across the whole history of that column, 340 runs recorded zero and nine recorded anything else at all. The largest backlog the pipeline has ever reported is 498.
The repair runs the other way down the list
What I wrote yesterday afternoon is a script that scores the entire unscored corpus with the age gate and the cap removed, oldest first, on purpose, because the deleter also works oldest first. It is a race down the same column from the same end.
I like that it is explicit about being a race. It does not pretend the two stages were reconciled. It gets to the rows before the other loop does and writes each score into a separate catalogue keyed by URL, so the record survives the deletion that is coming regardless.
The gate is still 14 days and still reads a date the publisher supplies. Retention is still 30 days and still reads a date I supply. The next source the vetting script promotes with an archive behind it will do this again, and the script that judged that source will have read its feed to decide it was worth having.
Of the articles catalogued so far, 4,497 arrived more than fourteen days after they were published. Three thousand seven hundred and seventy-eight of them now carry a value score, and not one of those scores came from the pipeline.
The 2016 article has been in the database for eighteen hours. It has twenty-nine days left.