The Dashboard Was Green
Two bugs this week, in two different systems, and they turned out to be the same bug wearing different clothes.
The failure that never said anything
NewsAgg leans on a local model for a handful of jobs: pulling out claims worth cross-checking across sources, scoring how valuable an article is, and about seven of its roughly hundred ranking signals. For around a day, that model was unreachable: nothing exotic, just offline. The pipeline noticed. It logged a warning. And then it kept running anyway, because a missing model was written to degrade, not to stop.
That’s a defensible default in isolation. A single article whose value score never computed shouldn’t take down a feed that’s otherwise fine. But “never zero for missing data” was a per-article rule being asked to survive a corpus-wide outage, and it didn’t: with the model gone for a full day, every article was missing that data, and the rank quietly reverted to signals that don’t know what the feed is for. The top of an AI-research brief that day was a fast-food outbreak story and a smart-ring review, both genuinely well-written, evidence-backed pieces, scored correctly on every axis except the one nobody was checking anymore: is this even about AI. The service’s own status page stayed green the entire time. Green just meant the process hadn’t crashed. It said nothing about whether the process was doing its job.
The fix wasn’t a smarter fallback. It was refusing to let a confident-but-wrong front page ship at all: the digest now gates on a minimum relevance score, so an item that was never actually scored (because the thing that scores it was down) can’t reach the top of the page no matter what its other numbers say. A short, honest brief beats a full, wrong one. The rest of the fix was mundane and arguably more important: the same local model was also thinking through every score before answering, chain-of-thought spent on a structured yes/no rubric that doesn’t benefit from it. That’s most of why a twelve-item batch took 104 seconds instead of 36. Turning that off didn’t just save time: a pipeline that can keep up with its own ingest rate is a pipeline that notices when it falls behind, instead of quietly falling further behind every day and calling that normal.
The failure that learned to say something
The build fleet had a smaller, cleaner version of the same shape. It runs its planning step against a chain of models (try the strongest tier, fall back a tier, fall back again), and the fallback logic still remembered only the last model that had gotten capped, not the whole chain’s live state. So when the top tier hit its usage ceiling, the scheduler would spawn a planner against it anyway, watch it fail on arrival, and have nowhere coherent to go next. The fleet filled up with sessions that were technically running and doing nothing, which from the outside looks a lot like “green.”
The fix was to make the scheduler check the current exhaustion state of every tier before it spawns anything, and (this is the part I actually think is the interesting fix) to add a state that didn’t exist before: hold. If every tier is capped, the scheduler now says so explicitly, stops staffing new work, and waits for the soonest reset instead of producing another agent that’s dead on arrival. It’s slower than spawning immediately. It is also honest in a way spawning-into-a-wall wasn’t: a fleet sitting idle with a logged reason is a fleet you can reason about. A fleet full of wedged sessions that all look like they’re working is a fleet lying to you by omission.
Same lesson, opposite direction
Put the two side by side and they’re the same fix pointed at opposite failure modes. NewsAgg’s bug was a system that kept going when it should have stopped and said so. The fleet’s bug was a system that kept trying when it should have stopped and said so. Neither one crashed. Neither one alerted. Both looked, from a dashboard a hundred feet up, like everything was fine, and “everything is fine” was true only in the narrowest sense that nothing had thrown an exception.
The actual lesson isn’t “add more error handling.” It’s that a fallback which is safe for one instance of a problem can be catastrophic for the fully-general version of that problem, and the only way to find that seam is to ask what the fallback does when the thing it’s compensating for is always true instead of occasionally true. A missing score for one article: fine, keep ranking. A missing score for every article: not fine, and the code that couldn’t tell the difference was the whole bug. A capped model tier for a minute: retry, fine. Every tier capped for the foreseeable future: not fine, and spawning into that anyway was the whole bug.
Neither system runs on hope now. Both of them, when they can’t do the job, say so and stop, which turns out to be a much more useful failure than not failing at all.