A setting reaching a container here has to pass four separate places:
docker-compose.ymlpasses it into the serviceapply-env.shon the server allows the deploy pipeline to set itapply-env.ymlputs it in the step environment- and actually writes it into the stream piped to the box
Nothing checked that all four existed. In one day, three settings shipped with a gap.
The loud one
POSTMARK_MAIL_FIFEROUTER_TOKEN was sent by the workflow and missing from the
server's allowlist:
refused: 'POSTMARK_MAIL_FIFEROUTER_TOKEN' is not in the deploy allowlist
The forced command rejects an unknown key rather than ignoring it, and exits before restarting anything. Loud, immediate, safe — the deploy failed and production was untouched. This is the failure mode you want.
It happened twice, because the fix was committed to the repo and the server was still running the copy it had been given by hand.
The silent one
OPERATOR_EMAILS was allowlisted, passed through by compose, and never sent.
Setting the secret in GitHub therefore did nothing at all. No error anywhere: the allowlist was not consulted because nothing arrived, compose substituted the empty default, and the operator console read an empty allowlist.
An empty allowlist admits nobody, which is the correct behaviour — so the console was shut for its own operator, correctly, for the wrong reason, with no way to open it. Two hours of looking at the wrong half of the problem.
It was worse than that: neither the compose line nor the allowlist entry had
been committed either. git add in the console commit listed explicit paths and
deploy/ was not among them.
Why the tests passed
The console has thirty tests about who may reach it. Every one of them monkeypatches the module attribute:
monkeypatch.setattr(admin, "OPERATOR_EMAILS", {"boss@example.com"})
So they exercise the decision and never the delivery. Every assertion
about authorisation was true. Nothing touched the path from secret to .env to
container to module, which was the part that did not exist.
That is a general shape worth naming: a test that patches a module attribute proves the logic reading it, and proves nothing about whether anything writes it.
What we built instead of being more careful
Eight static checks over the repository, comparing the four places against each other:
- every setting compose reads has a delivery route, or a documented exemption
- everything the workflow sends is allowlisted (the Postmark failure)
- everything the workflow sends reaches a container
- every sent value comes from a variable that is actually bound
- no allowlist entry is dead — the
OPERATOR_EMAILSfailure
Exemptions carry a reason and are themselves checked, because a bare entry is
indistinguishable from silencing the test. POSTGRES_PASSWORD has a test of its
own asserting it stays undeployable — that absence is load-bearing, not
incidental.
Then each of the three real bugs was reintroduced and the suite re-run, to see them caught rather than assume it. All three, by the check written for them.
The check that was wrong first
The guard failed on correct code the first time. It compared the key written to
.env against the step environment, and tripped on:
printf 'FIFEROUTER_SPEND_CAP=%s\n' "$SPEND_CAP_USD"
where the input and the setting have different names deliberately. It now checks the source variable.
Worth recording, because a guard that fires on correct code is a guard somebody switches off — and it would have been switched off by the same person who wrote it, within the hour, which is how these things actually die.
The underlying problem
Four places, no relationship between them enforced by anything. That is not unusual: env var plumbing is spread across a compose file, a CI workflow, a deploy script and application code by nature, and each is edited by somebody thinking about a different concern.
The fix is not discipline. It is one thing that reads all four and complains when they disagree — which is the same move as putting sign constraints in the database instead of remembering to pass positive numbers.