The demo that never shipped
An agent that nails the demo and dies in production is the most common failure I see. Here’s how I tell the two apart before anyone bets a roadmap on it.
A few years into this work, I stopped getting excited by demos. An engineer wires up an agent on a Friday, it handles a hard task on the first try, and the screenshots go in the deck. Then it’s October, the agent is switched off, and nobody wants to talk about why.
The demo answered one question: can this thing succeed? Production asks a harder one: what happens when it fails? People assume the distance between the two is just a little more testing. It isn’t. It’s a different kind of work.
What does the agent do when the model returns confident nonsense? When a CSM overrides a decision and, six months later, someone has to reconstruct why? The demo only walks the happy path.
So that is the work I focus on. Anyone can call an API. The real engineering is everything wrapped around it: the schema checks, the fallbacks, the override surface. That is what lets a revenue team hand a workflow to an AI tool and stop watching it.
Four questions I ask first
Before anything goes near production, I run the build through four questions. Miss any one of them and you have a demo. That is fine, as long as the room agrees that is what it is.
- What happens on the worst input, not the best one? “It probably handles it” is a demo answer. In production the behaviour on the bad path is designed and known, not hoped for.
- Who owns it when it’s wrong? Every automated workflow needs a name attached to it. “The agent did it” is not a sentence you can say to a customer.
- How would you find out it’s degrading? If the answer is a support ticket, you found out too late. The system has to watch itself.
- What does it touch that it has no business touching? Demos grab everything in reach. A real system pulls only the fields it needs, because every extra scope is one more thing you’ve signed up to defend.
I open with these instead of model choices or framework arguments, because they sort a project in about five minutes. They tell you whether you’re looking at something built for the stage or something you can put your name on. I only build the second kind. I’d rather have that conversation on day one than in October.