I save every prompt I type when I’m building something. I started doing it long before this blog, for a dull reason: I forget what I did, and the log helps me pick things back up. Going back through one of those logs recently, I found a session worth sharing. It’s closer to what building with AI actually feels like than a clean demo is.

The demos show you the good moment. You describe what you want, and a working app appears. That does happen, and it’s real. But most of the work isn’t that. A lot of it is more like this: a task that looked easy, and an AI that got it wrong a few times before we sorted it out.

The task

For the update I described earlier in this series, I had to work out which recordings in a new batch I already had. About 360 files had come in a zip; my server already held around 300. I just wanted the list of the new ones.

It looked like a one-minute job, so I asked plainly:

Compare the audio files in the zip against the ones already on my server, and tell me which I need to add. Use an AI model (DeepSeek) to do the comparison.

It gave me five different answers

The first count came back. Then it changed. Then it changed again — a different number each time it “rechecked.”

Six of the files on my server didn’t get passed to the model.

I don’t trust that this count is right.

At one point it stopped and wrote me something I didn’t expect:

I sincerely apologize. I’ve made repeated mistakes and kept correcting myself. I told you 234, then 160, then 88, then 85, then 95 — a different number almost every time.

Five answers to one simple question. I’m not saying that to put the tool down; it’s genuinely useful and I use it every day. But it’s the part the demos leave out. It was smooth and confident and wrong, and it didn’t know it was wrong. If I’d taken any one of those numbers on faith, I’d have loaded the wrong set of files into something I cared about, and nothing would have stopped me.

What actually sorted it out

What broke the logjam wasn’t a cleverer prompt. It was working out why the comparison kept failing — and that was something only I could know, because the files were mine.

No — the files on my server and the ones in the zip have been re-compressed to make them smaller. They’re not the same files anymore. You can’t match them as files. Compare them by title and date and work it out.

That was most of it. The model had been trying to match files as files, and failing, because the copies had been shrunk and re-encoded. It had no way to know that. I did. Once I changed the task from “match the files” to “reason about titles and dates,” it started to come right.

Then there was a second call to make:

This one is dated a year apart between the server and the zip. That’s a typo in the year — same recording, not a new one. Drop the duplicate. And don’t add a fuzzy date-matching rule for it; it’s a one-off, not a pattern.

Told to “handle date mismatches,” a model will happily invent a rule and start “fixing” things that aren’t broken. I had to say: this is an exception, treat it as one. Telling a one-off apart from a pattern is the kind of call I don’t think you can hand to the AI yet — at least I couldn’t.

After that the count settled — 92 files to add — and stayed settled.

What I take from it

I’m not an engineer, so I can only tell you how it looks from where I sit. But if this session taught me one thing, it’s roughly this: the AI does the boring work, and it does a lot of it, but it can be wrong in ways it can’t see — so my job is mostly to understand my own problem well enough to catch it.

Two things have stuck with me:

  • A confident wrong answer is harder to spot than an error message. “I don’t trust this” turned out to be a perfectly good prompt.
  • The useful move was usually a re-frame, not a retry. “Compare by title, because these are re-encoded” did more than twenty rewordings would have — and it came from knowing my own files, not from knowing how to code.

That fits something I wrote in an earlier post: the AI can handle the building, but the judgment still has to be yours. Hand it both and it’ll take you somewhere wrong, very politely.

A note on how this got written

I could rebuild that whole session — the wrong counts, the apology, the re-frame — only because every prompt I type is saved to a running log, tagged by project and date. I don’t keep a separate diary; the log is the diary. It’s also where posts like this come from. So, the honest version: I don’t have a great memory, so I just kept the receipts.

Key Takeaways

  • The “describe it and an app appears” demo is real, but it isn’t most of the work.
  • On a simple-looking task, the AI gave five different confident answers and flagged none of them.
  • What fixed it was knowing my own files — they’d been re-encoded, so they had to be matched by title, not as files.
  • Telling a one-off typo apart from a pattern is a judgment call worth keeping for yourself.
  • The AI does the work; the judgment stays with you. Don’t trust an unchecked number just because it sounds sure.