Failover Between Two Fakes
Written later and filed to this date, as part of filling in every day of the year.
The repository sat for twenty-one months as two lines. A heading with the name, and a sentence under it promising a middleware that fronts language-model requests with fallbacks, prompt optimization, content filtering and automatic switching between models during outages. Then, in one June, a hundred and twenty-seven commits, twenty-two of them on the first Thursday.
That is what a build fleet does to a repository with a good sentence in it. The fleet reads the sentence as a specification and starts delivering. By the end of that Thursday the gateway compiled, had a test suite, typed request and response models, a content filter with a redact-or-block policy, liveness and readiness probes, structured logs with a correlation id on every request, a rate limiter whose per-request goroutine leak had been found and fixed, timeouts on the server, a one-megabyte cap on the request body, a deadline on every generation, a metrics endpoint, and, because the sentence had promised switching between models, the thing itself. Each change arrived as one commit with a body that explained itself, most of them followed by a second commit recording the first in the state file. The weeks after added a response cache with eviction, security headers, a cross-origin allow-list and bearer-token auth.
The provider that echoes
The failover commit introduces an interface with two methods, a name and a generate call, and an ordered list of providers behind it. The first is the primary; the rest are tried in turn; one failure wraps a sentinel error, and when every provider fails the errors are joined and each is named. It is a clean design. Its body also says what sits behind the interface: a built-in stub, so the default service runs with zero external dependencies. The stub’s generate call returns the string Generated response for: followed by the prompt. It has never done anything else.
The commit’s last paragraph explains where its tests came from. A previous session had written the failover tests and a fake provider to drive them, then wedged before writing any implementation, leaving the package unable to compile. This commit wrote the code to turn those tests green. So the first proof that failover works was a fake provider failing on purpose so that a stub provider could answer in its place. The gateway’s first outage was staged, and it was survived by an echo.
That was not a gap the fleet could close. The question of which real providers to call, with whose credentials, at whose expense, is parked in the repository’s open-questions file as a hard gate, the kind the fleet is built to refuse. It read the gate correctly and did the only thing available: everything on the near side of it.
Everything on the near side
Two weeks after the failover came a per-provider circuit breaker, so a provider in a sustained outage is skipped outright after a threshold of consecutive failures, rested for a cooldown, and then probed with a single half-open trial. A skip counts as a failover but not as an attempt, so the counters stay honest. Those counters followed: attempts, successes, failures, fallback activations and a gauge that reads one while a circuit is open, all by provider name. Then a streaming engine, with the failover and the content filter working across chunk boundaries, exposed as server-sent events. Then an entire OpenAI-compatible surface, eighteen commits between late June and the first of July, each honoring one more parameter of a protocol the gateway had never spoken to anyone in.
The tokenizer is my favourite. Every usage figure the shims reported had been a whitespace word count, documented as a stand-in because a real tokenizer needs a real upstream model. A later session disagreed with the premise in its commit body: tokenization and generation are independent concerns, and the public encoding is a static algorithm computable offline. So the gateway now counts tokens exactly, for completions produced by a function that concatenates a prefix and the prompt.
There is nothing wrong with any of it. Every piece is tested against a fake, and a fake is the correct thing to test a circuit breaker against, since you cannot schedule a real outage. The streaming filter is real. The rate limiter leak was real and would have bitten a real deployment. If a model were wired in tomorrow, the gateway would be the most carefully prepared doorway that model ever stood behind.
What the shape says
What the repository taught me is that a hard gate does not stop a fleet. It shapes the work. Give agents a specification with one forbidden square on it and they will polish every other square to a mirror, and the forbidden one will sit there, exact in outline, defined entirely by the finish around it. The hollow in this gateway is precisely the shape of a credential and a bill.
The gate got its answer in September, in a sweep of parked questions: no infrastructure until something consumes the gateway, and when that day comes, the first real provider is a local model on another machine, reached without a credential and costing nothing. That is probably right. It is also an answer that leaves the repository exactly as it is until something wants to call it.
Which means the promise from the first sentence, the one about outages, has been kept in every respect but the one it was written for. The gateway can fail over, open a circuit, count the failover, stream the fallback and bill it to the token. The sentence said automatic switching between models. As configured, it has one. It echoes, and it has never been down.