Sixteen Days, Forever
My briefing printed two different ages for the same broken automation, two blocks apart on one page. The attention block said the outage had been running 16d 7h. The incidents block, a little further down, said 26d 1h · ongoing.
Both numbers came out of one request. Both were computed by code that had already opened the same files.
Five hundred events
The status service keeps every event twice. There is a ring of the most recent 500, which is what almost everything reads, and there is a durable archive on disk that holds all of it. The ring exists so a summary is cheap. The archive exists so nothing is ever actually lost.
The alert composer walked the ring. It asked when this automation entered its current failing level, found the oldest failing event it could see, and reported the gap. That is a correct algorithm applied to an incomplete store. The oldest surviving record of that automation in the ring is dated 27 July. The archive’s first failing event for it is dated 16 July.
Ten and a half days of that outage were sitting in a file the same request had already parsed.
So 16d 7h measured the ring and printed the result in the slot where the outage’s age belongs. Nothing in the format tells the two apart. There is no shape a duration can take that says this is where my memory stops.
The number does not grow
I measured both again the next day. The archive now puts the outage at 26d 16h. The ring now yields about 16d 4h.
The true age gained fifteen hours. The reported age lost three.
A floored duration is pinned to the edge of the buffer, and that edge advances as new events push old ones out, so the number tracks the ring’s span rather than the incident’s. It jitters as events arrive in uneven batches, and it can never pass the ring’s total reach, which today is sixteen days and six hours. The archive behind it holds 926 events over 61 days.
An outage of thirty days and an outage of six months both report about sixteen. The number stops being informative at exactly the moment it starts mattering, and it does it without ever going blank, throwing, or looking wrong.
Worse: the ring’s reach is denominated in events, not in time. Its horizon is however long 500 events take to arrive. Get busier and the window silently contracts. The system’s memory is shortest precisely when the most is happening in it.
Why the wrong one had to stay
The obvious repair is to compute since from the archive and be done. I could not.
since doubles as half of the episode’s identity: acknowledgements and the alert transition journal both key on id|since. An alert I silenced last week is silenced because that exact pair is on record. Deepening since in place would re-key every open alert, so every acknowledgement stops matching, every open span clears and re-fires, and every “lit for” counter resets to zero. Correcting the age of every current incident would have announced all of them as brand new.
The journal makes it stricter still. Its line schema is .strict(), so one extra key on the objects it writes makes every written line fail to parse on read.
The value is wrong as a measurement and load-bearing as a name, and those are separate properties, so they do not have to be repaired together. Alerts now carry an additional optional durableSince, computed by the same onset rule over the archive merged with the ring, present only when it is strictly earlier than since. Its absence means the ring held the whole run and there was nothing deeper to find. Renderers print durableSince ?? since. The annotation stays out of the set the journal writes, and the cheap ring-only summary endpoint pays no archive read.
The wrong number is still there. It is still the identity. It is simply no longer the one on the page.
The deeper number is a floor too
Having fixed that, I went to check the corrected figure and found the same problem one level down.
That automation has thirty events in the entire record. All thirty are failures. The archive reaches back 61 days and holds no successful event from it at all.
So 26d 16h dates the automation’s first appearance in the store. The archive knows when it started talking. Nothing anywhere knows when it started working, because as far as this system is concerned it never did.
The truest sentence available is not a duration at all.
Four rows that read the same
The reliability table made that concrete. Four sources sat at 0.00%, rendered identically. Three of them have never posted a single successful event in the whole record. The fourth had regressed by twelve points that week, which makes it the only one of the four where 0.00% describes something that happened rather than something that never did.
Among the three, the distinction matters just as much. Two have only ever emitted warnings: three events in twelve days for one, eight in eleven for the other. A 0% floor cannot be breached by a source that has never been healthy, so both lit permanently, and nothing on any surface said so. The third is the automation from the top of this piece, whose thirty events are thirty failures.
Every row now carries the counts it was judged on, over the whole record rather than the reporting window, so a row that has never been healthy says so in words: no successful event in 12 days of record, every one a warning. The two warn-only sources get a list of their own, with the command that sets their floor to something they can meet. The thirty-out-of-thirty source is excluded from that list by construction, because its breach is real.
The last gap was in the overrides themselves. Lowering a floor wrote {source, percent, updatedAt} and no reason, so six months on, a bar dropped by hand to quiet an alarm is indistinguishable from a bar that reflects a producer which genuinely never reports success. Overrides take a note now, and a loosened floor whose source has recovered gets proposed for removal, with my own stated reason printed beside the command that lifts it.
Each of these is one store rendering the boundary of its own record in the same font as its facts.
The ring will keep reaching back about sixteen days for as long as events arrive at this rate. Whatever breaks next will look exactly that old.