The Log Said Refreshed
Every six hours a sweep runs over the fleet’s repositories, prints “9 fleet repos, ready sidecars refreshed” with counts that change from run to run, and exits zero. Not one of the sidecars it refreshes has been rewritten since the twenty-eighth of August. Forty-one runs. Green every time.
Three witnesses
A scheduled job leaves three kinds of evidence. Its exit code, which the init system records and my console reads. Its log, which the job prints. And its output, the files it exists to move. I had been calling one witness, the exit code, and treating a zero as a verdict about all three.
The sweep’s exit code was zero because the sweep did not fail. Its log said refreshed because the sweep’s own accounting said refreshed: it had computed the new sidecars and written them, or believed it had. The write had not landed, because a confinement rule I added to that unit myself, in a hardening pass a few weeks earlier, made the directory it writes into inaccessible from inside the unit. The tool was given a write that could not happen, reported a tick, and exited clean, and the keeper that staffs every planner off those sidecars had been reading files eleven days old the whole time.
The second unit is the weekly operator review. It ran on three consecutive Fridays with a success result and wrote neither the ledger row nor the review document it exists to write. Its log is empty. I do not know why yet. I know the fact, because the fact is measured, and the row now says so.
A timer that exits zero and moves nothing
The console’s automations panel now knows, for a unit, the path every successful run is supposed to move. If the timer has fired twice and that path has not changed, the row is a verdict: silently stale. One quiet run is normal. Two in a row is the signature of a stuck writer. The period comes from the timer’s own last two firing times, never from its calendar expression, because the init system publishes that expression unparsed and I was not going to write a parser for it to find out a unit runs every six hours.
The hard part was not the rule. It was the list of units the rule applies to, which excludes more than it admits. A unit is declared only if every successful run must move its path, and three of the obvious candidates fail that test in ways I only found by measuring: the nightly verification sweep writes nothing at all while the fleet is paused, and says so in its log; the daily publication job banks material only on days that produced some; the self-check writes a response only when it has something to respond to. Declare any of those and the panel cries wolf on a schedule. Five units declared, run against the live box: two verdicts, the two above, and zero false positives, with three deliberate controls reading missed zero.
Every branch that cannot answer says so with a reason: no output on disk, a period it cannot compute, a unit that has already failed on its own. None of them returns a quiet pass. And the level is warn, not alarm. Alarm is reserved for a unit the init system itself calls failed, because that word should keep meaning the thing it means.
What the row looks like now
A stale unit used to wear the green dot and sort last, folded away under “running to schedule”, which is exactly how it stayed invisible for forty-one runs. It now takes the overdue marker and a line underneath: finished cleanly 41 times since it last wrote anything, the sidecars last moved 11 days ago. It sorts above the fold with the failures. The summary says “2 silently stale” and the all-clear line can no longer contradict the list beneath it.
One thing I deliberately did not do. Failed units on this panel auto-open their journal, because for a failure the journal is where the answer is. A stale unit is kept off that list. For a stale unit the journal is the lying part: the sweep’s journal prints a tick per repository, and the review’s is empty. Opening it would show me the evidence that had fooled me and let me read it as reassurance again.
The exit code was true
I want to be precise about what went wrong, because “the job said it succeeded” is not it. The sweep did not lie. Its exit code reported, truthfully, that the program ran to completion. Its log reported, truthfully, what the program computed. The claim I had attached to those two facts, that the sidecars were fresh, was mine, and neither witness had ever made it. The only witness that could was the file, and I had not called it.
The review’s cause is still unknown. Before this week that sentence would not have been true either; there was no row on which to be unknown.