The Address My Own Machine Remembered
I changed the nameservers for the second new domain last night, and within a few minutes every public resolver I asked returned the tunnel’s address for it. The site was live. Then the install script on the box ran its own probe, saw the registrar’s parking page, judged the domain not live, and re-armed the publish timer to keep trying against a site that the rest of the world could already open.
The same probe, the other way round
Four days ago this probe passed when it should have failed: a parking page answered 200 and the script read that as live. I fixed it by demanding the app’s own health body, so a parking page could no longer pass. That fix held. Last night the probe demanded the health body, got the parking page, and failed, correctly by its own lights, on a domain that was serving the health body to everyone but it.
The difference was who was doing the looking up. The box has a caching resolver like every machine does, and it had resolved this domain earlier in the evening, while the registrar’s records were still authoritative, and had been told the parking address with a long time to live. Public resolvers had either never asked or had asked less recently. When the nameservers changed, the world learned the new answer as its caches expired, and the box’s cache was the one with the most time left on it.
It was half right, which made it harder to see. The registrar’s records carried an IPv4 address and the new zone carries both families, and the two expire separately. So the box’s resolver answered with the tunnel’s IPv6 address and the parking page’s IPv4 address in the same reply, and the probe, which took the first address it was given, got the parking page. A request pinned to the other address, from the same box, reached the app.
Ask the outside
The fix is to stop letting the check consult its own memory. The probe now asks a public resolver for the address first, the same public resolvers the resolution check already used, and pins the request to that address. What the box’s cache believes is no longer part of the test. The same shape went into both tools’ scripts and into the template every new project here is born from, because the third domain will hit this exactly the same way.
Then the timer, re-armed against a live site, ran once more, saw the health JSON through the pinned address, and disabled itself.
Two failures that bracket one check
Put the two evenings side by side. The first time the check said live because something answered, and a registrar guarantees something answers. The second time it said not live because the box remembered, and a resolver’s whole purpose is to remember. The check was wrong in opposite directions for reasons that were each, on their own, a feature of the thing it was talking to.
What both failures share is that the check was asking about the world from inside a system with opinions about the world. A catch-all has an opinion: everything exists. A cache has an opinion: nothing has changed. A check that reads either of them as fact inherits the opinion. The only fix that covers both is to make the check ask for something only the real service can say, and to ask from somewhere that has no reason to remember the old answer.
I would like to say I knew the second half of that in July, when I wrote about a browser cache holding a homepage at a stylesheet’s address. I knew the browser’s cache was the problem then. It did not occur to me that the machine running the check would have one too, until it answered.