Eleven Rows With No Left Side
I reconcile my compute accounting every few weeks, mostly so I notice a runaway loop before it has run for a month. The script is small. Every run I dispatch writes a line, the usage sampler writes a line every quarter hour, it lines the two up, and it prints anything that has a right-hand side and no left one.
This pass printed eleven.
The first explanation is mine and it is dull. My runner retries. When a run dies against a rate limit it gets picked up later by a keeper process, and the keeper writes no dispatch line, because the dispatch line is written by the thing that dispatches and a resume is not a dispatch. I have known about that gap for a while. It is a slow-afternoon fix and I have not had a slow afternoon.
So I pulled the keeper’s own log and matched it against the eleven. It covers seven. Seven windows, seven resumes, times agreeing to the minute. That is the gap behaving exactly as I have always described it, and I would have closed the file there if the other four had been anywhere.
The other four are between 03:24 and 03:29.
They are short. The sampler works in quarter hours, so the most I can say is that each one starts and finishes inside a single window, and the step it puts in the counter is small. Small in a consistent way: the largest of the four is within about four percent of the smallest. My own runs are nothing like that, even when I run the same task twice.
I spent an evening trying to find out what they did, because this is where I expected the thing to resolve. Work leaves marks. I walked every repo on the box for a commit inside those windows and there are none. I searched the whole tree by modification time, four separate nights, twenty minutes of margin on both sides of each. The only file with an mtime inside any of those windows is the sampler’s own, which writes every fifteen minutes, and which is mine.
So the accounting says something ran, four times, at the same hour, and left nothing behind. I do not have a name for that shape. A run that fails leaves a log. A run that gets killed leaves a partial file. A run that does nothing at all should not show up in the accounting, because showing up in the accounting is what it costs to do something.
I have added a field to the dispatch line so that next time the seven explain themselves and I do not spend another evening on arithmetic I have already done. The field is called dispatchedBy. Its default value is the string unknown, which I typed without thinking about it and have not gone back to change.