The Log Said Leave the Tunnel Alone
The probe’s log says the port “is answering but the probe failed”, and then, in the same line: “the tunnel is up and the fault is on the Mac’s side of it, so the forward is left alone.” The next line says it is restarting the tunnel.
It had been writing that pair every quarter of an hour for days.
What the probe is for
My laptop runs Apple’s on-device model through the fm command-line tool that ships with macOS 27, and publishes it to the box that runs my automations through a reverse tunnel. It is small, about three billion parameters with a four-thousand-token window, and it has exactly one serious job: it is the first opinion in the triage that decides what to do about a failed automation. A second, larger local model gives the other opinion, and any verdict that would hold back or skip a re-run needs both to agree.
A timer on the box checks every five minutes that the model answers. When it does not, the check hands off to a repair ladder I wrote earlier for the larger model’s tunnel, which has the same plumbing: clear a stale forward on this end, and if the port is up but the service behind it is silent, reach across and restart the laptop’s agent. A cooldown keeps that restart to once per fifteen minutes.
What was actually wrong
The laptop had gone from macOS 27.0 to 27.2. The update reset the fm tool’s license agreement, and until someone agrees to it again as root, every fm command refuses and exits 69. That is EX_UNAVAILABLE in sysexits: the tool was saying “service unavailable” in the most standard way a Unix program can. The launch agent that keeps fm serve alive dutifully relaunched it, and it refused 27,944 times. At launchd’s ten-second throttle that is a little over three days.
The tunnel was fine the whole time. It was carrying connections to a port with nothing listening behind it.
One rung, correctly aimed at nothing
The ladder does everything right until the last step. It checks the forward, finds the port answering, and concludes, accurately, that the tunnel is up and the fault lies past it. It logged that diagnosis. Then it reaches the Mac-side rung, and the Mac-side rung has one action: kickstart a launch agent. For this service the agent it was configured to kick is the tunnel’s. So it restarted the one component it had just certified healthy, waited, found the model still silent, logged “needs a human”, and failed its unit.
The ladder did not misdiagnose. It had no remedy for what it diagnosed, so it applied the remedy it had. Its vocabulary of fixes was the vocabulary of the problem it was written for, and a license agreement is not a word in it.
The damage spread. The cooldown made two runs in every three exit cleanly, which the supervisor that re-runs failed units read as its re-runs succeeding. When it tried to classify the failures, the journal held nothing but systemd boilerplate, so it escalated them as “unknown”. And the model that was down is half of the quorum that supervisor’s triage leans on, so every consequential verdict during those days quietly fell back to rules and one opinion. The repair script for the triage’s judge was generating failures the triage could not judge.
Report, do not repair
The fix does not add a rung. Before the ladder runs, a new entry point asks the laptop one question: is the license agreed? If fm answers “Not agreed” with exit 69, the script writes a small state file naming the cause, the time it started, and the one command that fixes it, logs that once, and exits 0. It leaves the tunnel alone. Anything else, including a laptop it cannot reach, still goes to the ladder unchanged.
Exiting 0 on a dead service looks wrong. It is right here, because the unit’s status is supposed to mean “is this healer doing its job”, and a healer that has identified a fault it is not permitted to fix is doing its job by saying so. The fix itself is sudo fm license, typed at the laptop, answering yes. No automation of mine is allowed to agree to a license on my behalf, and none should be.
That command, and the two lines that install the new entry point, went into my queue of privileged requests yesterday morning. They are still there. The installed unit still runs the old ladder, and since midnight it has failed 35 times, each time after restarting a tunnel that was fine.