The previous posts are nine mistakes from a single stretch of work. Read individually they are ordinary. Read together they have a shape.
Almost none was caught by review, and almost every one was caught by something mechanical that had been added for a different reason.
What caught what
| The mistake | Found by |
|---|---|
| Deploy swapped a bind-mounted directory | A person looking at the site |
| Uptime monitor could not parse itself | Reading a log for another reason |
apply-env unparseable on main |
Trying to run it |
| Postmark token not allowlisted | The forced command refusing |
OPERATOR_EMAILS never sent |
The console staying shut |
| Suite testing its own ordering | CI, three times |
@types/node undeclared |
A clean container build |
| Strict flag with no off position | Re-reading the send block |
| Uppercase in an image reference | The org rename |
Two were found by looking. The rest were found by something that parses, resolves, builds or refuses — and in most cases that something existed for an unrelated purpose.
The most useful ones were secondary effects
actionlint went in because a dispatch-only workflow had been broken on main
for a day. It has since caught nothing, and that is fine — it converted a class
of bug from "discovered during an incident" to "discovered in CI", which is
value delivered whether or not it fires again.
The container build went in to answer which build is in production. Its first act was to find a dependency that had been undeclared since before anyone remembered.
PAC-902 — the product gate resolving every test reference — exists so intent
cannot drift from code. It caught renamed test classes twice in one day, both
times because a behaviour had changed and the acceptance criteria still
described the old one.
None of those were built to catch what they caught. Guards are worth more than their stated purpose, because the failures they find are the ones nobody predicted, which is by definition the interesting set.
The dangerous failures were all silent
The loud ones were fine. refused: not in the deploy allowlist cost ten minutes.
A container that will not start is unpleasant and unmistakable.
The expensive ones made no sound:
- a deploy that reported success and changed nothing
- a monitor that had never once run its assertion
- a secret that was set, allowlisted, and never sent
- tests that passed because of the order they ran in
Every one of those presented as working. The system was green, the logs were empty, and the only signal was somebody eventually noticing that a thing they expected had not happened.
Which suggests the question to ask of any new mechanism is not "how will I know if it breaks" but "what would it look like if this silently did nothing". If the answer is "the same as it looks now", that is the monitor to build.
Verifying the guard, not just writing it
The habit that came out of this: after building a guard, reintroduce the bug and watch it fail.
actionlint — put the empty expression back, saw it flag line 73, the same line
GitHub reported. The config-delivery checks — reintroduced all three real
failures, each caught by the check written for it. Then reverted, and confirmed
green.
It found something twice. The config guard's first version failed on correct
code, tripping on a printf where the workflow input and the setting have
different names deliberately. A guard that fires on correct code is one somebody
switches off — and it would have been switched off by the person who wrote it,
within the hour.
The uncomfortable part
Most of these were mine, made while writing careful code with good comments
explaining why it was correct. Several of the comments were correct. One of them
— the empty ${{ }} explaining why expressions must not be interpolated — was
itself the bug it warned about.
Care is not the mechanism. It is the thing you spend while the mechanism is being built, and the reason to build the mechanism is that care does not survive contact with a long day.