FifeRouter

3 September 2026 · mistakes ci

The workflow that was broken on main for a day, invisibly

An empty expression in a comment. GitHub parses comments. And it only parses a workflow when you need it.

$ gh workflow run apply-env.yml
could not create workflow dispatch event: HTTP 422:
failed to parse workflow: (Line: 73, Col: 14): An expression was expected

apply-env is the workflow that moves credentials to the server. It had been merged a day earlier, with every check green, and it could not run.

The cause

A comment. This one:

# Read from the environment, never interpolated into the script. A
# `${{ }}` expansion is substituted as *code* before bash parses it,
# so a value of `$100` becomes the expansion of $1.

Written to explain why a workflow input is read from env: rather than interpolated — a real hazard, and the note was correct.

GitHub's expression parser reads the whole file, comments included. ${{ }} with nothing between the braces is not a valid expression, so the file fails to parse. The comment warning about expression interpolation was itself an expression interpolation.

The part that made it dangerous

GitHub parses a workflow when it is about to run one.

apply-env is workflow_dispatch only, deliberately: credentials should move when a person decides they should, not as a side effect of a merge. So nothing parsed it after the merge. It sat on main in a state where every status check was green and the file could not execute.

And it would have announced itself at the worst available moment — the next time somebody urgently needed to rotate a key or push a fix to production. The failure was scheduled for whenever we were most in a hurry.

Why CI missed it

CI ran the tests, validated the product graph, and scanned for vulnerabilities. Nothing in this repository had ever read a workflow file.

That is easy to miss because workflows feel like they are being exercised — they run constantly. But the ones that run are the ones on push and pull_request. A dispatch-only workflow is code that nothing executes until you need it, which makes it exactly like a disaster-recovery script: the category of thing most worth testing and least likely to be.

The fix

actionlint in the gate, parsing every workflow the way Actions does. Pinned by version and installed from the project's own script rather than a third-party action — this repository's secrets include a deploy key, and a linting step is a poor reason to add a new trust relationship.

Then I checked it actually catches this, rather than assuming. Reintroduced the empty expression into a copy, ran the linter:

apply-env.yml:73:4665: unexpected end of input while parsing ...

Line 73 — the same line GitHub reported. Exit 1. Reverted, and all workflows clean.

That verification step is not ceremony. A guard you have not seen fail is a guard you are hoping about, and this one was being added specifically because hoping had just cost a day.

The comment, rewritten

The explanation was worth keeping, so it stayed — with the example removed and the reason it was removed added underneath:

(And do not write an empty expression in a comment to illustrate the point: the expression parser reads comments too, and an empty one fails the whole file at dispatch time — which no CI job here parses, so it stays invisible until the day you need to deploy.)

Which is now the most useful sentence in the file, and the only one written by the bug it describes.

The generalisable bit

Ask, of anything that only runs on demand: what parses this, and when?

For a dispatch-only workflow the answer was "GitHub, at the moment you need it". For a rollback script it is often "nothing, until the outage". For a migration's down-path it is "nothing, ever, in most shops".

Those are all the same shape, and the fix is the same shape too: make something routine parse the thing, so the first execution is not also the first syntax check.


← All posts