Safe to Restart, Having Read Nothing
For about ten minutes a watchdog of mine restarted three services in a loop. Nothing was wrong with any of them. They had started refusing anonymous readers half an hour earlier, which was the entire point of the afternoon’s work.
A fourth service got three restart remedies inside an hour, then a hold and an escalation, for the same reason.
What I had actually changed
The status service on that box had 67 routes and 48 of them answered any caller who could reach it. Calling that a status page undersells it: those routes carry the live event ring, the durable archive behind it, every derived view over both, and every store I have typed into by hand, including acknowledgements, mutes, maintenance windows and incident notes. One route was already behind a bearer token, which was decorative, because nearly everything inside what that route returned was separately readable with no credential at all.
So the reads went behind a credential. On the same afternoon, several browser-facing services on the same box went behind a shared credential gate.
Both changes worked on the first try. Neither of them is what this is about.
The monitor’s idea of healthy
The watchdog keeps a registry: one entry per service, each with a probe and an expectation. Most entries read / and expect a 2xx. Several carry remediate scope, which means the watchdog may restart what it watches without asking anyone.
GET / now answers 401.
401 is not 2xx, so four correctly gated services read as wedged processes, and the watchdog did the one thing it is authorized to do about a wedged process. A monitor is only as good as its idea of healthy, and mine had a very specific one that it had never been asked to say out loud.
The fix that would have been worse
Widen every expectation to accept 401. One line per entry, the red goes away, and I was most of the way through doing it before I noticed what it would leave me holding.
A 401 comes from the gate. The gate runs before any routing, so it answers whether or not the service behind it can serve a single byte. Accept 401 as health across the board and that registry stops watching services and starts watching doormen. Every entry stays green as long as the lock works, with the room behind it dark.
So each entry got repointed at the thing it is actually responsible for. Two of the services gate in-process and exempt a liveness route, so their probes moved there, where a 2xx proves the app itself is answering. Two others sit behind the gate as a separate proxy process, so each got two entries: one crossing both hops, one crossing only the first. A failure between them now names which of the two to act on, which the single merged entry never could.
The two that would never have gone red
The loud failure took ten minutes to find because it was loud. Underneath it were two checks that would have gone on passing.
The status service’s own repo carries a command that runs before any guarded restart of it. It probes 33 read paths, parses what comes back, and reports whether the stores look intact enough to bounce the process. Its failure test tolerated a 4xx.
Gate the reads, and all 33 of those paths answer 401. 401 is a 4xx. Every probe passes. The command prints SAFE TO RESTART with full confidence, having parsed nothing whatsoever. Its certainty would have peaked at the exact moment its knowledge hit zero.
The media library has a probe that checks whether its catalog scan is current. It read the catalog route, which is now gated, and it fails safe on any response that is not ok. No data, no complaint. It would have reported freshness as unknowable, forever, while watching nothing, with no symptom except a check that never again had an opinion.
Both of those are the restart loop with the alarm taken off. The loop was a gift: wrong in the direction where someone comes running. A probe that reads nothing and passes is wrong in the direction where nobody does.
What green had been measuring
Those probes had been green for months. Every green tick was the same measurement, repeated every 45 seconds: this service answers a request that carries no credential. That was true. It was also, precisely, the property I spent the afternoon destroying.
The monitoring had been reporting the exposure continuously, in real time, at high resolution, under the heading of health. There was no blind spot here. The instrument was aimed straight at the hole for months, printing the reading in green.
The watchman with no key
The sequel arrived the next day, once the probe had been taught that a 401 is a refusal rather than a death.
The status service does what it should with a refused read: it counts it. read_no_credential, logged like every other auth failure. That counter exists so a consumer I forgot to hand a credential to shows up as a rising series instead of as silence.
The probe ran every 45 seconds. Roughly 1,900 log lines a day, every one of them a healthy monitor knocking on a door it is not carrying a key for. I sampled a quiet three-minute window and found four refused reads. A three-minute window holds exactly four probes. All four were mine. The first thing the new signal did was bury itself under my own watchman.
The probe moved to the one route left deliberately anonymous, and the counter went back to meaning what it was built to mean.
Tests green, tile dark
A projects console reads a summary from the status service to fill one tile. The obvious repair was to hand it the token every writer already holds. Tests passed. In production the tile stayed dark.
A hardening pass from weeks earlier had listed that credential file as inaccessible to that unit. The confinement was right and my repair was wrong: the token it wanted also grants write, and a page that renders a summary tile has no business writing anything. The console already loads the gate’s secret, because it runs a gate of its own, so it now mints a read-only cookie from that instead. Nothing the process could not already do, and no exception carved into the confinement.
The difference showed up in exactly one place, against the live socket. Every test I owned passed on the wrong answer.
The window
One entry in that registry is still different. It watches a dev server with nothing exempt, no liveness route, nothing at all that answers without the gate in front of it. There is nowhere to move its probe. So its expectation stays 401, that 401 is written down as its healthy answer, and the entry watches a lock and admits it.
Every other service on the list now keeps exactly one route that will answer anybody who asks. It is the route the monitor reads. The only part of the system I can still see without a key is the part I decided not to lock.