The deploy reported success. The files were on the server — you could cat
them. The site served the previous build.
Nothing was broken in any way a check could see, which is what made it worth writing down.
The swap
The webroot is a directory the edge bind-mounts. Updating it looked like the safe pattern: unpack the new build beside the old one, then swap.
rm -rf public.new && mkdir public.new
tar -xzf - -C public.new
mv public public.old
mv public.new public # ← here
Atomic, reversible, no window where the directory is half-written. It is the right shape for a swap and it is wrong here for one reason.
Inodes
A Docker bind mount resolves the path once, at mount time, and then holds the
inode. Rename the directory out from under it and the container does not
follow — it keeps serving the directory it mounted, which is now called
public.old.
So after the swap:
- the new files are in
public, which nothing is reading; - the old files are in
public.old, which Caddy is serving; lsshows exactly what you expect, in the place you expect it;- every health check passes, because the site is up. It is just old.
The failure is invisible from every direction except viewing the site and noticing a change you made is not there. Which, if the change was subtle, you might not.
Replace contents, not the directory
rm -rf public.staging && mkdir -p public.staging
tar -xzf - -C public.staging
test -f public.staging/index.html || { echo 'incomplete build'; exit 1; }
mkdir -p public
find public -mindepth 1 -delete # the mounted directory stays put
cp -a public.staging/. public/
rm -rf public.staging
The mounted directory is never renamed, moved or recreated. Its inode is the same one the container mounted at start-up, and its contents change underneath.
Note the test before anything is deleted. Emptying the live webroot and then
discovering the tarball was truncated is a worse afternoon than the one this
post is about.
The thing we actually changed
The fix is four lines. What took longer was accepting the reason the original was wrong, because the original was written by someone applying a good habit — atomic swaps really are the correct pattern for replacing a directory that nothing has mounted.
The habit was right. The context had one property the habit does not account for, and no amount of care about the swap itself would have surfaced it.
That is why the fix ships with a comment explaining the inode, not just the code. The next person to touch this will know that atomic swaps are correct, because they are, and will reach for one for exactly the same good reasons.