Skip to content
Notes

Why software built with AI breaks after the demo

2 min read

Somebody builds a tool in a week that used to take a team a quarter. The demo works, everyone’s impressed, and it goes into use. Three months later nobody trusts what it produces, and someone has gone back to the spreadsheet without telling anyone.

I build custom tools for companies when nothing off the shelf fits, from company intranets to scanners for recruiting firms, and AI coding tools are part of how they get built. The failures I’ve seen, my own early ones included, come from how the work gets organized.

Long sessions drift

A model gets less reliable as its working memory fills. An hour in, it forgets a decision it made earlier, writes a second copy of a function under a new name, or refers to a file that doesn’t exist.

Bigger context windows made this easier to miss, because sessions can now run for hours past the point you should trust them. So the first thing I decide on any build is where each session ends, and I cut the work into pieces small enough to finish well before that.

Whatever wrote the code can’t review it

A model checking its own code brings the same assumptions it wrote it with, so it finds the work sound. It’s why you can proofread your own paragraph five times and keep missing the same word.

Software teams fixed this long ago with code review. Plenty of people building with AI dropped the habit because code arrives faster now, which is the reason to review more of it. For me that means a model from a different company, with different blind spots. A second model from the same company mostly catches typos.

What the review is for

I review for three problems that testing won’t catch.

  • Drift, where the build wanders from what was asked, one reasonable-looking decision at a time.
  • Overbuilding, like a settings system for two settings nobody will ever change. Every extra part is something to work around the next time the tool needs a change.
  • Choices that won’t hold up once next month’s features go in, which is usually what forces a rewrite.

My first builds gave one model too much scope, and they fell apart once people used them every day. Now each piece gets built, reviewed by a model from another company, then read by me, and I keep the pieces small enough to hold in my head. I still read every plan myself.

What it costs

Building this way takes three to five times the effort of the fast way, and it’s still cheaper than paying for a rebuild once people stop trusting the tool.

Four questions before someone builds you a tool

  1. How is the work broken up, and how big is each piece?
  2. Who or what reviews the code, and is it something other than what wrote it?
  3. What happens when you ask for a change six months from now?
  4. Who keeps it running if the builder moves on?

What’s the tool your team keeps saying it could build in a weekend?