When a deploy changes nothing visitors can see
The fix shipped, the origin served the new version, and everybody kept getting the old one. How caching turns a successful deploy into a claim you cannot verify.
We shipped a visible fix, confirmed it was being served correctly, and watched visitors keep getting the broken version. Nothing errored. Nothing appeared in any log. The page was simply old.
Two reasonable defaults, one bad outcome
Modern frameworks mark statically generated pages as cacheable for a very long time. On its own that is sensible: the page is static, so cache it.
Caching layers in front of a site generally respect what the origin tells them. Also sensible on its own.
Together they meant the cache had been told to hold pages effectively forever, and it obliged. Deploying changed what the origin served and changed nothing anyone experienced. There was no error to find, because nothing had failed.
How to spot it in seconds
Request the public URL and look at the response headers, not the page. Anything saying the response came from cache, together with an age measured in thousands of seconds on a page you just deployed, is the entire diagnosis.
Checking the origin is the instinct and it is the wrong move: the origin was never wrong, which is exactly why it wasted our time.
The rule we settled on
Long caching is correct when the name of the thing changes with its content — a hashed asset filename can safely be cached forever, because a new version is a new name.
It is wrong for anything served at a stable address, which is every page on a site. There, the cache must revalidate, and private or personalised responses should not be shared-cached at all.
A cache that never expires turns every deploy into a claim you cannot verify.