The month we shipped nothing and every light stayed green
For a month, every merge to this site built clean, passed checks, and never reached production. Nobody noticed, because the pipeline only lied at the very last step.
In late June I was reviewing this site, the one you’re reading, and something felt stale. A feature I clearly remembered merging weeks earlier wasn’t on the page. I checked the repo. Merged. Checked the build. Green. Checked the preview deployment for that merge. Beautiful. Then I listed the production deployments, and there it was: the last one that actually went live was from May 28. Production had been frozen at PR #228 for a month while roughly twenty merges piled up behind it, every one of them building successfully as a preview and going nowhere.
Here is how you engineer that failure. You wire production deploys through a CI job that fires a deploy hook after tests pass. Reasonable. Then your GitHub Actions minutes run out, and every workflow run starts dying in about seven seconds with a billing error, before a single step executes. The deploy hook never fires. Meanwhile the platform keeps doing its own thing on every push: it builds a preview, comments a link on the PR, everything looks alive. The one signal that mattered, a production deploy with your commit on it, was absent. Absence doesn’t page anyone.
The part that stings is that I had receipts for everything else. Test counts, accessibility sweeps, screenshot audits of preview URLs. All of it real, all of it verifying the wrong artifact. A preview build proves your code works. It says nothing about whether a user typing the domain gets it.
The fix was deleting infrastructure
The repair, PR #248, was two moves. Turn the platform’s native auto-deploy back on, so merging to main deploys production with no custom machinery in the path. And delete the CI deploy job entirely rather than fixing it, because a deploy path that can fail silently is worse than no deploy path at all. CI still gates the merge with lint, types, and tests. It just no longer stands between a merged commit and the live site.
Since then, one habit: after anything ships, I check what production is actually serving. Not the dashboard, not the workflow summary. The deployed thing itself, with my own eyes or a script that fetches the live page and greps for the change. It costs thirty seconds. It would have saved a month.