Graded Harder Than It Was Built
Written later and filed to this date, as part of filling in every day of the year.
The first commit is dated the Fourth of July and contains a license, a gitignore and a README one line long. The second commit is eleven months later and brings in 9,916 lines under the subject bring existing application under version control. Between the two there is nothing in the log, which is the only account the repository gives of its first year: an application was written, ran somewhere, and was not tracked.
The application is an earthquake forecasting dashboard. It pulls events from the USGS catalogue for a region (the default is a box around California, magnitude 5.5 and up), and after every large quake it publishes a best guess for the next one and keeps a tally of how close the guess came. The README says that in its tagline and then, a few sections down, says what the model is.
Sixty lines
The forecast is a Poisson waiting-time model, and the whole of it fits in one function. Take the events above the threshold, sort them, compute the gaps between consecutive ones in days, average the gaps. The forecast for the next event is the last event plus that average. The 95% interval comes from the exponential distribution’s quantiles at the same rate. The function returns a dictionary with a model name, Poisson-GR v1, and the sample size.
The README’s limitations section says the model assumes earthquakes occur randomly in time and “may not account for clustering”. That sentence came in with the import. Earthquakes cluster; that is what an aftershock is.
Then the fleet arrived
From the June commit on, the log is the work of coding agents in numbered sessions: eighteen commits on the first day, sixty on one Sunday ten days later, the session count past eighty by the end of the month. The model function gained one guard in all that time, for a region whose events share a timestamp, so the mean gap is zero and the rate would otherwise go to infinity and collapse the interval to a point. Much of what the sessions built after the validation and the plumbing was measurement.
The accuracy endpoint came first: mean absolute error in days, the median, the hit rate inside a thirty-day tolerance. Then a signed bias, so a region could be seen to forecast early or late. Then a skill score against a naive baseline, so a forecast could be shown to beat guessing or not. Then a Brier score on the within-horizon probabilities, then calibration of the 95% interval, then a reliability diagram, then all of it per region and at seven, thirty and ninety days. The scorecard page grew a column for each. The sixty-line forecast was by then wrapped in several hundred lines of instruments for saying exactly how wrong it was.
The assumption, measured
Three sessions in late June did the thing the README’s one line had been waiting for. The first computed the dispersion of the waiting times, the coefficient of variation, and classified each region’s catalogue as clustered, quasi-periodic or Poisson. The second joined that classification to the scorecard and showed what followed: clustered regions forecast worse. The third fit a Gamma renewal interval and placed it beside the Poisson one on the dashboard, with the ratio of their widths, as a comparison. None of the three changed the forecast the product issues.
That boundary was deliberate and written down. Replacing the headline model moves every golden test for the forecast, the interval and the probabilities; it changes what the product says to a person reading it. The sessions filed it as a question for a human and kept building beside it. The repository ends up with an exact, graded, per-region account of how much its one assumption costs, and the assumption still in place.
What it taught
A forecast is cheap and a grade is expensive, and a fleet that may build anything additive will build grades. That is not a complaint. The instruments are real; the Brier scores and reliability bins are the kind of thing a person writing the model alone would have promised in a README and never shipped. What the agents could not do, by the rule they were working under, was the single destructive change the instruments pointed at. So the repository is a precise description of a decision nobody has made.
The dashboard card the first of those three sessions added is titled “Is the Forecast’s Poisson Assumption Valid?” The README answered that in one line, three weeks before anyone built a card to ask.