The Dead Code Was Running
My deploy checker printed DEAD CODE next to a security fix I had committed that afternoon, and it was wrong.
Two probes settled it. Both came back with behaviour the older build could not produce: one response missing a header that the old code set unconditionally, one request refused by a guard that did not exist nine days earlier. The service was running the fix. The tool whose entire job is watching that service said it wasn’t.
A commit describes a repository
The service runs straight out of a working tree. Nothing compiles, nothing gets copied into a release directory. The process starts, reads the source files sitting on disk, and serves them.
Its health endpoint reported a commit sha. Specifically, it reported whatever .git/HEAD said at the instant the process booted, and I had been reading that for weeks as “the code that is running.”
Those are two different questions, and this was the day they gave different answers. The working tree got the new files first. I restarted six minutes later. The commit landed two minutes after the restart. So the process loaded that afternoon’s code while reporting a commit from nine days earlier, and my checker did exactly what I built it to do: compared the sha the service reported against the sha in my checkout, found a nine-day gap, and told me the fix was not live.
I nearly restarted the service to deploy something it was already serving.
Then I worked out that the error runs in both directions, and the other direction is the one that would have hurt. Boot with a dirty tree while HEAD happens to be current, and the tool prints running HEAD, no drift over a process loaded from uncommitted files that no commit describes. A false alarm costs a restart. A false all-clear is a machine running code that exists in no history.
The fix is unglamorous. At boot, take a sha-256 over the source files the process actually loads (about a megabyte across a hundred and two files, four and a half milliseconds) and put that digest in the health response. The checker digests its own checkout the same way and compares. Matching digests mean the running program is byte-for-byte this checkout, whatever the commit list claims. Differing digests mean real drift, proven instead of assumed. A process too old to report a digest at all gets an explicit unverified rather than a confident verdict in either direction.
The sha was never lying. It was answering a question about my repository, and I had been reading the answer as a statement about my program.
What the new instrument found
Once the comparison worked, the news got worse.
On the 27th the live process was running a commit from the 17th, with the tree standing twelve commits and one entire deploy ahead of it. Ten days. Everything in that gap had been written, tested, committed, pushed, and was serving nothing: an alert for a budget burning too fast, one for a backup going stale, one for archive lag.
And a self-backup worker. I built that specifically so the data directory would stop depending on my remembering to copy it, and it had spent ten days undeployed, which is to say the data directory had spent ten days still depending on my remembering to copy it. The thing built to delete a manual step had become a manual step, and I had already crossed it off.
The health endpoint returned 200 that entire time. Overall status: ok. Nothing was down. Everything was fine, in the narrow sense that everything the instruments were looking at was fine.
Three more of the same substitution
Once you have seen one instrument answer an adjacent question, you find others.
The headline metric of this whole project is time reclaimed. On /stats there is a published method and a deliberately empty row where the number goes, because a figure with no method behind it is marketing.
The internal service was printing a figure anyway. It read one from an environment variable: a number I typed into a config file and then forgot, rendered in every summary as the live KPI. The history store held 322 recorded points. All 322 are the same value. Not clustered. Identical, because the recorder copied whatever the summary was holding and the summary was holding a constant.
Two days ago I switched that recorder on by default. It captures every five minutes, so the next restart would have started writing 288 points a day of a hand-typed number into the north-star trend, and the dashboard and the glasses display would have drawn it as a line. The line would have been perfectly flat. A flat north-star reads as stability, not as absence. Nothing on any surface could have said the difference.
I am not printing that number here, and it isn’t modesty. It was never a measurement, so there is nothing to publish. The empty row stays.
The one-line status summary I glance at most often said 2 ok, 1 degraded, 1 outage. Both of the ok sources had gone silent: one had posted nothing for six weeks, the other for two and a half. Neither was named on the line. Silence had entered the tally as health, because the summary counted rows and a dead source’s last row still says ok.
You cannot check a process from inside it
So I added an alert for exactly this failure. Compare the digest this process loaded at boot against the sources on disk now, and if they differ, say out loud that the running code is stale. It works. It is also constitutionally incapable of catching the case that started all of this.
That alert compares the digest frozen at its own boot, so it can only ever indict the process it is running inside. When my briefing command composes it, the composing process is a fresh one I launched seconds ago from the current tree, which is by definition not stale. The briefing renders the current source’s derivation over the data directory on disk, then prints it under a header that reads like a report on the running service.
It is not a report on the running service. It never was. And a process old enough to be badly stale is old enough to predate the alert entirely, so the sicker the patient, the quieter the symptom.
There was a cruder version of the same hole underneath it: because the briefing reads the data directory off disk, a completely dead service still rendered a full green briefing that never once mentioned the service was down.
The only repair was to stop asking the code and start asking the machine. The briefing now makes a single loopback request to the live service on a one-and-a-half second budget and classifies the reply with the same core the deploy checker uses, so the two can never disagree. A failed probe never fails the briefing; it just has a name now.
And when the answering process turns out to be too old to report a digest, the briefing falls back to comparing commits and prints probably STALE. Not stale. Probably. That is the strongest thing a report can say about a machine it only inferred.