run(db.execute(
"TRUNCATE ledger, stripe_events, api_keys, sessions, accounts "
"RESTART IDENTITY CASCADE"))
Every test starts from an empty database. Reasonable, fast, and correct until somebody adds a table.
Three times in one day
page_views. Added for analytics, not added to the list. Two tests failed
in CI:
assert s["total"] == 3
E assert 4 == 3
Four, because an earlier test's view was still there. Those assertions were not about analytics at all — they were about which tests had run first.
subscribers. Added for the mailing list, not added to the list. Seven
tests failed with IndexError on an empty outbox, and one failed asserting
pending and getting confirmed. Both symptoms of a subscriber surviving into
the next test: the second subscribe found an already-confirmed address and
correctly sent nothing, which is the no-enumeration rule working perfectly and
looking like a bug.
The rate limiter. Not a table at all — process-global memory. Every test
shares one client address, so after twenty sign-ups the rest got 429. Harmless
until sign-up became rate-limited, then load-bearing, and which tests failed
depended on collection order.
Why it kept happening
The list and the schema were two descriptions of the same thing, kept in step by memory. Nothing enforced the relationship and nothing could — there is no natural place for "and also add it here" to be checked.
The failures were also the worst kind to debug: they blame the wrong code. A
test asserting 4 == 3 looks like an off-by-one in the counter. A test finding
confirmed where it expected pending looks like a state machine bug. In both
cases the code under test was correct and the previous test was the fault,
which is not where anybody looks first.
And they only appeared in CI, because these tests need a real Postgres and the laptop did not have one reachable — so the loop was: push, wait, misread the failure, look at the wrong file.
The fix that stuck
Stop keeping the second description.
async def _truncate_everything(db_module) -> None:
rows = await db_module.db.fetchall(
"SELECT tablename FROM pg_tables "
"WHERE schemaname = 'public' AND tablename <> 'schema_migrations'")
names = ", ".join(f'"{r["tablename"]}"' for r in rows)
await db_module.db.execute(f"TRUNCATE {names} RESTART IDENTITY CASCADE")
Ask the database what tables exist. Adding one needs no corresponding edit, and forgetting is not a failure mode that exists any more.
schema_migrations is excluded, because emptying it would re-run every
migration on the next connection.
The other kind of shared state
The limiter needed the same treatment for the same reason — it is process-global state that a test must start from nothing — but it could not be discovered, so it is one explicit line with a comment saying why it is there.
There is a lesson in the asymmetry. Tables are enumerable, so the fix generalises. In-memory singletons are not, so each one is a thing somebody has to remember, and the honest response is to have as few as possible rather than to be better at remembering.
The one that was not this bug
The same day, trial credit moved from sign-up to email verification, and every test assuming a zero starting balance broke. That looked identical — assertions off by a constant — and it was a different problem: the behaviour had changed, not the isolation.
The fix there was not to patch thirty assertions but to make the grant off by default in tests and explicit where it is the subject. A test about the ledger should not also be a test about the grant, or changing the grant amount edits thirty unrelated files.
Two failures with identical shapes, two different right answers. Which is mostly an argument for reading the failure rather than pattern-matching it, having just spent three rounds pattern-matching the previous one.