Get Camino build log · 7 of 10
Tests encode beliefs, and beliefs can be wrong at scale
Everyone brags that the robots test their app. The part that matters more: a test suite can be perfectly green and loyal to the wrong answer — and four of mine were.
The robots test my app: sixteen end-to-end journeys against the live staging site on every deploy — “twelve tests, twelve seconds” when the suite first landed. It grew. That’s what test suites are supposed to do.
Sign-in runs through the real magic-link machinery — no test backdoor, because a test-only backdoor would be testing the backdoor. The suite’s very first run caught a React hydration error firing on every page load. Humans had scrolled past it for days.
That’s the part every AI-build post brags about. Here’s the part that matters more: tests encode beliefs, and beliefs can be wrong at scale.
When a family tester found the engine giving a Spanish citizen the foreign-national registration form, we didn’t just fix the bug. We audited every rule in the catalog against 181 generated profiles and 16 classes of expectation. Fifteen findings in one night.
The uncomfortable four: four existing tests were asserting bugs as correct behavior — green checkmarks faithfully protecting the wrong answer. One test persona, modeled on the exact household that reported the bug, had the bug baked into its expectations. The suite wasn’t lying. It was loyal to what we’d believed when we wrote it.
So the audit became infrastructure: the 181-profile matrix now runs in CI on every push, and any fix that changes behavior starts by grepping the tests for assertions that encode the old behavior.
Today: roughly 4,100 lines of test code against 14,500 of application code. The AI wrote most of both, in minutes each. Deciding what the tests should believe — and re-interrogating those beliefs when reality disagreed — that stayed human.
Hand-testing stops scaling exactly when the product starts working. So does trusting your own test suite.