Four Hundred Out of Four Hundred
Four hundred tickers went out and four hundred came back. That is the line in the log that killed the study.
Nothing timed out. Nothing came back empty. There was no retry loop to write, no gap to interpolate, no list of failures to work through in the morning. In every other data pipeline I have built, 400 of 400, 0 missing is the day going well.
What the machine is for
The engine is a factor research tool, and its product is the verdict rather than the backtest. Anyone can produce a curve that goes up. The hard part is deciding whether to believe one, so the whole thing is built around nine gates that a result has to clear before it gets called certified. Seven of the nine are mandatory, and a mandatory failure blocks the verdict instead of printing a warning under it.
Until this week it had only ever run against worlds I made up: a null world with no edge in it, which it rejected on five mandatory gates, and a synthetic world with a real 8%-a-year premium in it, which cleared eight of the nine. Those are useful because I know the answer in advance, and limited for exactly the same reason.
So this was the first study on the actual market. Free path: as-filed quarterly fundamentals straight from the SEC, 72,248 point-in-time rows over 7,202 filers, joined to daily bars from Yahoo’s public chart endpoint. 400 companies, 1,196 trading sessions, 469,798 price rows. Five minutes and twenty-one seconds end to end.
The numbers that wanted to be believed
The backtest was good. Sharpe 1.45. Compound annual return 22.6%. Worst drawdown 14.9%, recovered in 106 days.
Those are the numbers you would put in a deck.
They are also almost entirely the market. Against an equal-weight benchmark of the same 400 names, the strategy’s active return was 1.17% a year with a tracking error of 8.5%, which is an information ratio of 0.113. The regime_consistency check put it plainly: the best single year accounts for 465% of the total active return, so every other year together is negative against the benchmark. One good year is carrying the record and the rest is drag.
A 22.6% return with an information ratio of 0.11 is a report about a rising market that happens to have my signal riding on it.
Zero is the finding
None of that is what blocked the verdict. The first gate to fail was data_integrity, and it failed on two audits.
The first is survivorship. Across the 4.8 years of price history, 0 of the 400 companies stopped trading. Attrition: 0.00% per year. The audit wants to see at least 0.5% a year and notes that real universes lose names to bankruptcy, merger and delisting at something like 2% to 8% a year.
The second is universe_includes_losers, which looks at the 393 names the strategy actually held at some point and asks how many of them later went dark. None of them did. The audit puts it in its own words:
Not one name the strategy ever selected went on to die. That is not how markets work; the universe is being built from survivors.
The fetch statistic and the disqualifying evidence are the same number, read twice.
The filter ran before the first request
Two independent mechanisms put the survivors there, and only one of them is Yahoo’s fault.
Yahoo will not serve you a company that stopped trading in 2011. The adapter says so in its own docstring, declares survivorship_free = False, and ends with a line I wrote and then had to take my own advice from: nothing here stops you looking, it stops you believing.
The other mechanism is mine. The candidate pool is the largest revenue reporters that still carry a ticker, and it maps company to ticker through the SEC’s current company file. That file lists companies that exist today. So the pool was filtered down to survivors at selection time, before a single price request went out. Even a perfect price vendor would not have saved this run, because the losers were gone by the time I went looking for prices.
I thought I was testing whether a signal picks good companies. I was testing it on a list assembled from companies that turned out fine.
Nine minutes
The gate thresholds live in a policy file committed to git at 15:19 UTC. The study ran at 15:28.
Nine minutes buys no virtue. It buys an ordering the machine can check by itself: the verdict carries policy_pre_committed: true because the run compared its own start time against the file’s commit. Without that, every threshold in the file is something I could have chosen after seeing the result, and a bar you can move is not a bar.
That file includes min_track_years: 10.0. The sample is 3.6 years. I committed a threshold I already knew this data could never clear, because the alternative is arriving at 3.6 years of history and deciding that 3.6 years is enough.
What refusing cost
The verdict came back REJECTED, and the data audits were not the end of it. Six of the seven mandatory gates failed. Deflated Sharpe 52.4% against a 95% bar, once the 25 recorded trials are accounted for. Probability of backtest overfitting 65.5% against a 50% ceiling, across 924 train and test partitions, with the winning variant averaging 0.06 Sharpe in sample and 0.01 out. The leakage probe shuffled the signals and found that noise beat the real thing 26.9% of the time.
The deflated Sharpe gate did the arithmetic on what would settle it: significance would need 2,633 years of track record. I have 3.6.
Two gates passed. Turnover came in at 2.0x a year against a 3.0x ceiling, and drought survival passed because the worst three-year stretch left me 2.8% behind the benchmark, which is easy to sit through in a record where the strategy barely departs from its benchmark at all.
There is an escape hatch. A human can record having looked at a specific failure and accepted it, and the run manifest keeps that on file. Using it here would have been the only interesting way to fail.
The strategy is not rejected. It is unjudged, and it stays unjudged until I pay for a price file that contains the companies that went under. The free one will keep returning four hundred out of four hundred.