NewsAgg

live · case study · Open the app ↗

This page is a full write-up: architecture, decisions, failures, and evidence. Source status: write-up only: the code stays private; this page is the public artifact. Withheld: Feed infrastructure, deployment identifiers, and a private relevance pass stay out; architecture, prompts, failure history, and measured results are documented below.

An always-on AI-news aggregator: 42 feeds in, local models ranking what matters, cross-source disagreement surfaced as a trust signal. The flagship case study: architecture, prompts, failures, verification, and measured results.

Most aggregators flatten the news into one headline. This one clusters articles about the same event across outlets, ranks what is actually worth a reader’s time with local models, and surfaces where the sources disagree. It is also the project I can document most completely, so this page is the case study the rest of the site keeps promising: what it is, how the agents built it, what broke, and how I know it works.

The problem

I wanted a daily AI-news brief that (a) doesn’t show the same press release fourteen times, (b) tells me when outlets contradict each other instead of averaging them into mush, and (c) runs entirely on hardware I own, with no cloud LLM calls. No product did all three, and the third constraint rules most of them out by construction.

The pipeline

The backend runs a staged pipeline on a schedule. Every stage is skippable and restartable, because half the failure stories below are about stages failing quietly.

1. Ingest: pull 42 RSS feeds, fetch full article text.
2. Embed: sentence-transformers MiniLM, 384 dimensions, L2-normalised, stored as raw float32 bytes in SQLite. No vector database.
3. Cluster: two articles join the same story when cosine similarity exceeds 0.75 AND they published within 72 hours of each other; connected components do the rest.
4. Claim extraction (local LLM): for clusters with two or more distinct sources, extract falsifiable claims, mark contradictions and corroborations between them, and score the cluster’s disagreement.
5. Value triage (local LLM): every new article gets a research-value score and topic, single-source ones included.
6. Signal scoring: roughly one hundred independent scorers (97 registered at last count) roll up into seven facets and one merit score, with a receipt explaining it. Only 7 of the ~100 signals call the LLM; the rest are pure CPU.
7. Reputation + ranking: sources earn a decayed dispute-rate reputation; final rank is quality × source tier × reputation × freshness, where quality is the geometric mean of merit and value.
8. Retention: old clusters age out whole, never article-by-article.

The LLM work runs on Ollama with mid-size open models (a 27B dense model for claim extraction, a 35B mixture-of-experts for triage). One tuning note that surprised me: turning the models’ thinking mode off tripled throughput on batch triage (104s down to 36s on a 12-item batch) with no measurable quality loss on this task shape.

How the agents built it

This repo is the clearest fingerprint of the fleet workflow the site is about. 248 commits over about two months; 187 of them carry agent co-author trailers, several distinct Claude models among them. I steered, reviewed, and merged; the agents wrote.

The design decision that made parallel agents workable: one signal per file, discovered by decorator at import time. There is no central registry file to edit, so five agents can each add a scorer concurrently with nothing to merge. Most of the codebase’s growth (the signals engine is about 29k of the backend’s 38k lines) happened this way.

Representative prompts

The two prompts doing the heaviest lifting, abridged:

Claim extraction (system prompt):

You are a precise factual claim extractor for news articles. A claim must be falsifiable: “The battery lasts 18 hours”, not “The battery life is good”. A newer article superseding an older one is NOT a contradiction. Output valid JSON only.

Value triage (after the regression described below):

STEP 1: is this item about ARTIFICIAL INTELLIGENCE? The test is NOT “would an engineer find this useful”. It is “is this ABOUT AI”. If it is OUT OF SCOPE, score 0.0-0.1 and STOP. A brilliant non-AI paper is a 0.0 for this reader.

That second prompt earns its bluntness; see the next section.

What failed

The honest part. Each of these shipped as a real fix commit, subject lines quoted verbatim.

• Silent degradation, the flagship failure. The whole pipeline is designed not to fall over when the LLM endpoint is unreachable. On 2026-07-14 that resilience became the bug: with triage down, ranking quietly fell back to merit alone, which is topic-blind by construction, and the front page filled with well-evidenced stories about sea-worm jaws and a fast-food outbreak. Three fixes came out of the postmortem: “fix(ranking): merit is not relevance, and a firehose must not own the front page”, the triage prompt above, and self-reporting (each pipeline run now records its LLM health, and the read surfaces disclose how many recent items are unscored instead of pretending the ranking is complete).
• fix(ingestion): full-text extraction has been silently broken for EVERY article. The extractor had been failing quietly and every downstream stage had been reading teasers, not articles. Found by reading output, not by an alert; the alert now exists.
• fix(llm): an outage must not be stamped as poison. Early versions marked articles as processed when the LLM call failed, so a one-hour outage permanently unscored an hour of news. Transport errors and unusable answers are now distinct, and only the latter consumes the item.
• fix(ingest): stop cutting arXiv abstracts to a third before the engine reads them. A truncation bug meant the ranking engine judged papers on a third of their abstract.
• fix(signals): one failure seen from two facets is punished once, not twice. Independent scorers can observe the same underlying defect; naive summing double-counted it.
• fix(pipeline): drop unsourced LLM claims instead of mis-attributing them. When the model produced a claim it couldn’t tie to a source, the old code guessed. Guessing attribution in a tool about source disagreement is self-defeating; now it drops them.

How I know it works

• 654 backend test functions across 42 files; the test suite is slightly larger than the application it tests (36k vs 38k lines). Frontend adds ~204 pure-logic cases.
• Every registered signal passes a contract test: it never raises on ten hostile inputs, stays in [0,1], is deterministic, and satisfies an anti-gaming rule (appending 2,000 words of filler may never RAISE any signal’s score).
• The TypeScript wire types are generated from the backend’s Pydantic models, with a drift test that fails if they diverge.
• Ranking pins: cross-source dispute gating, pagination tiebreaks, and both silent-degradation failure modes are pinned by regression tests.
• Tests run with no model, no network, and an in-memory database, so they actually run everywhere.

Measured results (as of 2026-07-24)

• 42 feeds configured (27 AI, 11 tech, 4 science; 22 primary sources, 17 journalism, 3 aggregators)
• 10,172 articles ingested, 260 active clusters, 1,983 extracted claims
• 56 claims currently marked disputed across sources; 261 with counted corroboration
• 733 pipeline runs recorded; about a fifth of historical runs failed partway, which is exactly why every stage is restartable
• Source reputation currently holds 10 sources healthy and 33 on probation, with autonomous demotion wired but nothing demoted yet

What is withheld, and why

Per the site’s standard (full intellectual transparency, selective operational transparency), this write-up excludes: network identifiers, hostnames, ports, and tunnel details for the machines involved; credentials and token mechanics; the deploy workflow’s target identity; and one entire subsystem, a second private LLM pass that scores each article’s relevance to my own project portfolio, because its config is literally a map of my private repos. The architecture, decision log, prompts, failure history, and numbers above are the reusable part; none of the withheld material changes how you’d rebuild this.