Fifty-One Minutes After It Answered
Written later and filed to this date, as part of filling in every day of the year.
The monitor for the house battery lost its gateway partway through a long Wi-Fi outage, once enough polls had failed in a row, and posted an error. The gateway answered again on the very next poll. Fifty-one minutes after that, the tile on my dashboard still said the gateway was unreachable. Nothing was unreachable. The monitor had simply never been given a sentence for the moment things come back.
How a level sticks
My status feed holds one level per producer, and the level is whatever the producer’s most recent event said. The summary walks backward from the newest event for as long as the level repeats, and the oldest event in that run is when the level began. That is the whole rule. There is no decay, no expiry, no second opinion. A producer that posts an error owns that error until it posts something else, and the dashboard’s health view and the heads-up display on my glasses both read the level off the feed and color accordingly.
The rule is right. A level that faded on its own would hide a fault that nobody had fixed. But it puts an obligation on every producer that is easy to miss when you write the failure path first, which is the only order anyone writes them in: whatever posts the bad news must also post the good.
Three in one week
The battery monitor was the third case. The first was the same monitor a few days earlier, reporting an alert from the gateway named SystemConnectedToGrid. That alert names the normal state. The house is attached to the grid; this is not news. But an alert is an alert, and the monitor’s batch handler posted every one at warn, so a yellow row sat on the health view for three days describing the system working.
The second was my tabletop game assistant, which drives a virtual tabletop through a bridge and posts an error when the bridge goes down and self-healing gives up. The bridge came back, as it does, and the heartbeat that noticed wrote one line to the session transcript. The feed heard nothing. The bridge heartbeats every ten seconds, so for more than a day the feed said “bridge down and self-heal exhausted” about a bridge that was answering six times a minute.
Then the gateway outage, where the recovering poll zeroed its failure counter and pushed nothing, and the tile stayed red until some unrelated alert happened to emit an event and replace the level by accident.
Three producers, one omission. Each had a path that said “this is broken” and no path that said “this is fixed”, because fixed is the quiet state and quiet states do not feel like they need a sentence.
The fixes, and their shape
The fixes are small and all the same. Informational alerts post at ok, and a mixed batch takes the level of its notable members. The heartbeat’s recovery block posts the ok that matches the error above it. The poll that finds the gateway back queues a recovery as the first event of its run, so a genuine fault the same poll discovers still gets the last word and keeps the latch; and only an outage that crossed the alert threshold earns a recovery event, so a quiet poll stays quiet.
The last one carries the subtlety. Ordering matters in a feed that keeps only the newest level: the recovery has to come first in a run, not last, or it would overwrite a real fault found seconds later. Pairing the events is the rule; the order they go out in is where the rule meets the clock.
To clear a level that is already stuck, the honest move is not to delete it. It is to post one corrective ok through the producer’s own helper, with a message that says what it corrects, so the feed’s history shows a monitor that was wrong and then said so.
The rule
The charter line that came out of this, written into the workspace instructions when the feed gained a third lane for states only I can change, reads: “warn/error is for breakage an agent or a rerun can clear, and every non-ok path needs a recovery ok.”
I had three producers that could say “broken” and could not say “better”. The fifty-one minutes were not the monitor failing. They were the monitor doing precisely what it had been given words for, and no more.