Do Not Make Them Consistent
The glasses said outage for 18h 49m. One source in the list beneath it had been failing for 9h 58m.
The digest, served by the same box in the same minute, had that source failing since 2 September and the overall verdict red since the middle of July.
I have written about this bug before. That is the part worth telling.
The essay that exempted it
In August I published Sixteen Days, Forever, about a status service that keeps its events twice: a ring of the most recent 500, which is cheap and which almost everything reads, and a durable archive, which holds all of it. An alert measured how long a failure had lasted by walking the ring, and so reported the ring’s reach as the outage’s age.
The fix in that essay added a deeper field for alerts and left one face alone on purpose. The sentence is still on this site: “the cheap ring-only summary endpoint pays no archive read.” I wrote it as a feature.
That essay’s last line was “Whatever breaks next will look exactly that old.” I meant it as a warning about the ring. I did not notice it also described the face I had just exempted.
The rule
Eight days later the ring bit harder. The roll call of every source the service knows was being built from it too, and by then the ring reached back only about a day and a half, so 7 of 16 sources had no row at all. Two alerts had been journalled as cleared though neither source had ever posted a success. Their failing events had aged out. Those two false “cleared” lines are still in the log, on purpose: the record of the fix says they stay.
That fix moved five faces onto the durable record. Then it wrote a boundary into the repo’s plan, in bold: the summary and its siblings “stay ring-only ON PURPOSE”, ending with “Do not ‘make them consistent’ in either direction.”
It gave two reasons. The summary is the hot endpoint the glasses poll, so it must not pay an archive parse on every request. And its record type is copied by hand into two clients, so a field that changed meaning would ripple.
Both were true that day. Read now, the first reason contains a selection rule. The face was exempted because it was the most read. The number I consult most often was the one chosen to be cheap, and cheap meant bounded by a buffer that shrinks whenever the system gets busier.
The reasons left and the rule stayed
On 2 September the archive learned to parse each file once per change instead of once per read. A warm archive read now cost about two milliseconds. The first reason was gone, and nothing in the note said what would retire it.
Three days later a session taught the summary to say how long the overall verdict had held and when it was last green. On 25 September another taught it to name which source was holding the verdict, and since when. Both were careful, tested features about duration. Both measured over the ring, as the plan said to.
On 27 September a session lined the glasses’ list up against the digest: 19 sources against 34. Two of the missing were in an error state. The ring had trimmed them, so the face I read without deciding to read was hiding precisely what it exists to show. That session restored the missing rows, wrote in its commit that neither of the note’s premises applied any more, and then amended the note’s heading to say its duration half “still stands”.
It found both reasons dead and repealed one clause.
The second reason ran backwards
The next evening’s session took the rest. Every duration on the summary now comes from the archive merged with the ring. The cost was answered by a cache keyed on the files’ fingerprints, not by an exemption: a poll between writes pays a few stat calls, and the full recompute, about 110 milliseconds, runs once per change to the store.
The second reason fell in that commit message, and it fell backwards. The note had kept the summary shallow so that one field would not carry two meanings. By then every other face already served the durable start time. The summary was the second meaning. For five weeks the rule had been guarding the inconsistency it named as the harm.
What the glasses say now
outage for 74d 5h. The source that read 9h 58m reads 26d 7h.
Seventy-four days back from that evening is 16 July. “Sixteen Days, Forever” went up on 12 August, four weeks into an outage it could not see from the face it left alone.