We built an external uptime check because a monitor on the box cannot report that the box is gone. It ran every fifteen minutes and failed every single time, from the moment it was written.
Twenty-seven runs, twenty-seven red. The service was healthy in all of them.
The line
assert d["concept"] == "moneypath.verify", f"classified as {d[\"concept\"]}"
An escaped double quote inside an f-string expression. On the runner's Python
that is a SyntaxError, and the whole script dies before the first assertion.
The probe had already done its work by then. Look at the log and the correct answer is sitting right above the traceback:
{"concept":"moneypath.verify", ... "eliminated":{"vendor-frontier":"residency"},
"chosen":"infra-large"}
File "<string>", line 4
SyntaxError: unexpected character after line continuation character
The routing decision was intact every fifteen minutes for three days. Nobody could tell, because the thing checking it could not start.
Why it was invisible
It was not invisible. It was a red X, on schedule, four times an hour.
That is worse. A check that has never passed is indistinguishable from a check that is broken in a way you have decided to live with, and after a day of it being red, red is what it looks like. Everyone learns that this workflow is noisy and stops opening it — which is a rational response to a signal with no information in it.
The failure mode is not "the alert did not fire". It is "the alert fired constantly and therefore meant nothing", which is the same outcome reached more expensively.
Why the tests did not catch it
There were none. It is a shell script embedded in a YAML file, running a Python
one-liner via python3 -c inside a single-quoted argument. Three languages
nested in each other, and the only thing that parses the whole stack is running
it.
CI did not catch it either, for a reason worth being precise about: CI validates
product intent and runs the test suite. Nothing in this repository read a
workflow file until we added actionlint — and that came later, after a
different workflow turned out to be unparseable on main for a day with every
check green.
The fix, and the fix that mattered more
The line became:
concept = d["concept"]
assert concept == "moneypath.verify", "classified as " + concept
No f-string, no nested quotes. Ugly, and it parses.
But the durable fix was making it impossible to ship again. actionlint now
runs in the gate, parsing every workflow the way Actions does — and I verified
it against this specific bug rather than assuming, by reintroducing the escaped
quote and watching it flag line 73, the same line GitHub itself reported.
What it changed about how we build monitors
Two habits came out of it.
A new monitor must be seen passing before it counts. Not "the workflow ran", but "the assertion evaluated and was true". The first green run is part of shipping it, not something that happens later on its own.
A red check that nobody investigates is worse than no check. If it is going to be red for a known reason, it should be off or fixed. Leaving it red trains people to ignore the class of signal, and that training generalises to the checks that are telling the truth.
The monitor has run green ever since, and the first thing it caught was something real. Which is the whole point, and would have been three days earlier.