Two Events, One Instant
Written later and filed to this date, as part of filling in every day of the year.
What. My status service writes a briefing every morning. One of them said an outage had been lit for four days and twenty-one hours, flagged as longer than usual, the usual being under a day. Four lines further down, its incidents block said the same condition had been ongoing for thirty-nine days and twenty-one hours. It had never once cleared. And in the week’s incident record, sixteen of thirty-seven incidents had lasted zero milliseconds, while the alerts list showed the source behind them lit for days.
When. On the page, by reading two blocks of one briefing against each other. Each block was consistent with itself; the contradiction existed only for a reader who looked at both.
The cause. Two writers, one shape. The alert journal, when an alert’s derived onset moved while the alert kept firing, recorded a cleared and a fired for the same condition in one call, at the same instant: seventy-three such pairs in 348 rows. The code that folds the journal into spans read each pair as a recovery followed by a new episode, and shredded one unbroken outage into thirty-eight short ones. Its baseline for what counts as usual was built from those fragments. Separately, one monitor posts a warning and an all-clear for each of its alerts, stamped at the identical instant, on every poll. The summary sorted the pair and took the last: degraded. The incident detector walked the same pair in ascending order and let the all-clear close the run, after zero milliseconds. Same two events, opposite readings, both shown.
The fix. Two rules, one idea: events at the same instant cannot be put in order, so nothing may depend on their order. In the journal, a cleared and a fired for one condition at one instant is a re-key: the span continues under the new key with its original onset, and no closed span is recorded. In the incident record, a source’s level at an instant is the worst level it reported there, the tie-break already used everywhere else that levels are combined. On the live data the journal went from 181 episodes to 108. The incident archive went from thirty-seven to twenty-one, zero-duration incidents from sixteen to none, and the monitor’s availability from 71.43 percent to 0.30, still ongoing. Its onset moved back by three days, to within fifty-seven milliseconds of the onset the alert journal had recorded all along.
What guards it now. The incident rule lives in its own module, shared by the summary and the incident detector, so they can no longer read one pair two ways; the journal’s onsets and its baseline of what is usual now come from a single fold of the log. The stored journal was not edited; the residue heals on read. And the commit names the counts that must not move, a deploy-drift condition with forty-seven genuine episodes among them, so the next change to the fold can prove it repaired the ties without swallowing real churn.