The Only Thing Deleting Rows
The dry run printed archive 140 -> 135 and I read it as a rounding detail.
It was the tool telling me, in advance and in plain numbers, that it was about to destroy five facts.
The service is a small API my automations POST events to. Recent events live in a ring that holds 500 and then forgets; underneath it sits an append-only archive, the durable record that outlives the ring. A row aging out of the ring is housekeeping. A row in the archive is a claim that something happened, and a restore has no business un-saying it.
Restoring a backup was un-saying five of them. The restore stage overwrote the live archive file with the copy inside the artifact, so every event archived after that backup was taken simply stopped existing. The repair pass that was supposed to catch this couldn’t: it backfills from the artifact’s own ring, which by definition cannot contain a row archived later. Two halves of the same system running with opposite signs (one closing the gap forward, its sibling opening one backward), and the delta line reporting the net as unremarkable arithmetic.
That was 06:12. By 07:43 I had found three more.
The rest of the ninety minutes
The backup was destroying the restore point it replaced. The artifact filename was date-only and the writer was a bare rename with no existence check, so a second backup on the same day silently overwrote the first, and the documented off-box command, which writes to a fixed path, overwrote on every single run. The README asserted that the newer artifact is “strictly better.” It is not, and the README documented three counterexamples itself, one of them four lines above the claim. Artifacts are composed through loaders that drop rows the current schema rejects, so a store that has just gone bad composes to an emptier artifact: the file gets worse exactly when the data is most at risk. I reproduced it: a two-event restore point replaced by a zero-event one, and a full restore point replaced by an artifact the CLI itself printed “is NOT a full restore point” while writing it.
The restore was duplicating the archive it merged. The backup composer reads the archive as one flat list (active file plus every rotated sibling), but the restore stage rewrites only the active file and never touches a sibling. So it wrote the merged list back whole, every sibling row landed a second copy, the sibling kept the original, and each round-trip landed another. Three real events became four archive rows, then six, then eight. One event recorded four times after three round-trips. The dry run double-counted identically, so it reported the inflation to me as a calm archive 2 -> 3.
A restore stage that failed part-way destroyed the file it was replacing. Two of the nine store writers rewrote a whole file with a plain write call, which truncates before it writes a byte, so a write that died mid-flight (a full disk is the cause the failure report itself names) left the file emptied. A truncated log doesn’t even surface as an error, since readers tolerate a torn trailing line. It just reads back as fewer rows.
The part that actually frightened me
That last bug had a second half. The partial-failure report (the thing that prints when a restore dies mid-way, the thing you read while your hands are shaking) told the operator that a re-run is safe.
The re-run was not safe. The archive stage derives its input by re-reading the active file, which it had just truncated. So the “safe” re-run re-read an empty file, wrote the artifact’s rows alone, exited 0 with no warning, and resurrected the first bug of the morning: the one that deletes everything newer than the backup.
I reproduced that too. An archive holding events A and B, one torn write, then the recommended recovery step, left A alone. Exit code zero.
The recovery tooling wasn’t merely broken. It was giving confident, specific, wrong advice at the exact moment I would have been least equipped to question it.
What the four had in common
I went looking for the usual suspect and didn’t find it. There was no gap where a check should have been. The suite was green through the entire life of the first bug, and the backup tests asserted the precise invariant it violated: that a healthy archive is a superset of the ring. Green all the way through, asserting the right thing.
Across a whole day of digging, nothing else in that service was losing data. Not the ingest path, not the workers, not the ring rotation, not a crash. Every row that ever went missing went missing at the hands of the backup-and-restore machinery. The one subsystem built for the express purpose of surviving a bad day was, empirically, the only agent of the bad day.
The reason is boring: I had never restored. Backup ran, exited clean, produced a file of plausible size, and I filed that as working. The artifact had never been opened, never fed to the parser that would eventually have to read it. The whole apparatus was a write path with an assumption bolted to the end.
The failure is structural, not careless. You exercise ordinary code by using it. Exercising recovery code means staging the disaster, and nobody stages the disaster, so the recovery path is the one part of a system whose first real execution happens on the worst day you will have.
Read it back
Every fix converged on the same move, and it isn’t “write more tests.”
Backup now re-reads the file it just finished writing, through the exact parser that restore uses (not a validator I wrote for the backup’s own convenience, the real one), and confirms every store’s row count survived the round trip. A failed verify exits non-zero and refuses to record the export stamp, so a corrupt artifact never gets to masquerade as a good backup and the staleness nag keeps nagging.
The pre-restart preflight got the same treatment twice over. It used to boot the service once against a copy of the stores, which proves the new code can read what the old code wrote. But at boot the new code’s own workers rewrite those stores, so the shape the service actually re-reads on its next restart was never exercised at all. A writer whose format its own reader rejects would pass, then crash-loop on the second real restart. And the write rehearsal counted raw rows on disk, which proves the bytes landed but not that the service’s endpoints accept what its own writers produce. Different schemas, here, for the same event.
So the preflight now boots the sandbox, lets it write, and boots it a second time against the stores its first boot wrote. Then it asks the still-running service, over its own live endpoints, for the event it just posted, and fails the verdict if either endpoint declines to hand it back.
It starts the thing twice to find out whether it meant what it said the first time.