Every Test, Five Times Over
Written later and filed to this date, as part of filling in every day of the year.
The repository began as the smallest thing a site generator will make for you, one page and a README that still says “Seasoned astronaut? Delete this file.” Twelve days later it was a history reader: articles drawn from public archives, a search index in a worker, a reading page with a drop cap and a Continue button, curated paths you can share by token. It had a test suite. And for a stretch of its life, the suite could not be run.
The record
The state file from that stretch is a document written under duress. The box is “slow-partial,” it says: the shell and git are fast, but starting a node process takes about four minutes, so the test runner and the build are “impractical.” Two single-file runs each produced zero output over eight to ten minutes and had to be killed. An open question titled “fleet-wide node-startup hang” records a survey of the machine: other projects’ tooling was hung too, the supervisors were healthy, and the best guess was resource exhaustion from many agents launching node at once. A later journal entry says the hang had worsened from node-only to “a total shell wedge,” every command stalling at the shell’s own profile. The sessions were doing what careful sessions do. They measured, they wrote down what they saw, they declined to build on a machine they could not verify on, and they left.
What they saw was real. The box was contended. But it was not why the tests would not run, and the thing that was why had been sitting in the repository the entire time.
What the runner found
The test runner had no configuration file. Without one it uses its default pattern for finding tests, which walks the whole tree. Inside the repository was the harness’s own directory, the one the agent tooling keeps its settings in, and inside that were worktrees: full copies of the source left behind by earlier sessions that had asked for an isolated checkout and never cleaned up. Each copy carried every test file. The runner found each test four or five times over.
Each duplicate gets its own worker process. On a loaded machine the pile of workers takes longer to answer than the pool’s sixty-second patience, so the pool gives up on each of them in turn, and the summary at the end reads: Test Files, no tests; Errors, thirty-seven. No test had failed. No test had run. Thirty-seven workers had been declared missing, and the symptom, a long silence followed by a report of nothing, looks from the outside exactly like a machine that cannot start node. The misdiagnosis held for several sessions because it was half true.
The fix is fourteen lines: a configuration file that tells the runner to look in the source directory and never inside the harness’s directory. With that in place the suite ran clean, eighty-nine tests passing, on the same contended box the state file had written off.
What it taught
Two things, and the second is the one I keep.
The first is about the harness. The tooling that runs an agent leaves artifacts inside the repository it is working on, and a repository is a tree that other tools walk. The same day’s commits found the harness directory untracked but not ignored, one git add away from history, and ignored it; and the first commit of that day had untracked a symbolic link to the dependencies directory that an earlier isolated session had committed without noticing, and which made every command in the main checkout fail with a loop error until it was gone. The harness is a tenant. It has to be fenced off from every tool that scans the tree, or it becomes part of the project in whichever tool scans first.
The second is about diagnosis under a true but incomplete explanation. Several sessions had a reason for the silence, the reason was observably real, and so none of them read the one line the runner printed at the end. “No tests, thirty-seven errors” is not what a hung node looks like. A hung node prints nothing. A runner that got far enough to count thirty-seven errors had started, found files, spawned workers and lost them, which is a different failure with a different cause, and the number was on the screen the whole time. The commit that fixed it shows what the others did not do: it read the output as the output of a test runner rather than as evidence about the machine.
The same commit tried the obvious alternative, running files one at a time instead of in parallel, and reports that it made things worse: a sixty-second fail-fast became a slow serial march of timeouts eight times as long. It left the residual flakiness on the record as what it was, contention from too many agents at once, and not something a configuration file in this repository could address. That sentence is the boundary of the fix drawn in the fix, and it is the part the next session would otherwise have had to rediscover.
The astronaut line is still in the README.