Here’s a problem that sounds harder than it is. New files arrive on the cutting board every hour. I sort them in passes, over several days. The automatic check keeps copying. So how does the same photo never get imported twice?

No database. One small ledger, plus a cheap shortcut, do the whole job.

The Ledger Does the Real Work

When the hourly check copies a file onto the cutting board, it writes one line in a small record — a plain text ledger I’ve been calling R throughout this series.

The key is the important part. It isn’t the file name alone — it’s which phone the file came from plus its path on that phone. Two different photos that happen to share a name (and phones do reuse names) can’t be confused, because they came from different places. Each line also carries a state:

  • copied — it’s on the cutting board, waiting for me.
  • transferred — it went into the archive.
  • excluded — I looked and chose not to keep it.

Before copying anything, the hourly check reads R: if this file is already in the ledger, skip it — whatever its state. And the ledger is append-only. Once a file is written, its line is never deleted, only updated. So a file is handled exactly once, forever. A reject never comes back to offer itself next hour; a keeper is never imported twice.

That last part — skipping the excluded files too — is what most tools forget. It’s easy to not re-copy something you kept. The subtler win is to also never re-show something you already looked at and passed on. Without it, every rejected photo would drift back onto the cutting board every hour, and sorting-over-days would be impossible.

The Cheap Shortcut

Reading the ledger for thousands of files every single hour would get slow. So there’s a fast first cut before the ledger is even consulted: a simple timestamp line — everything older than a fixed date is ignored immediately. The ledger is the source of truth; the timestamp is just there to make the common case quick.

(My separate, older gallery tool uses an even simpler version of the same instinct: before copying a file into the archive, it just asks the file system, “is this already there?” and skips if so. Same idea, no ledger — and fine for a tool where I do everything by hand.)

The Honest Weak Spot

I’ll say plainly where this is fragile, because “no database” has a cost.

The ledger is the single source of truth for “handled.” If R ever got corrupted, or got out of step with the real files, the skip logic could misfire — re-offering something, or hiding something it shouldn’t. A real database would give me transactions and integrity checks that a text file doesn’t.

For my scale, I decided that risk was smaller than the cost of running and syncing a database I’d have to babysit. If R breaks, the worst case is I re-sort some photos of my daughter — annoying, not catastrophic, and nothing is ever lost because the backups are untouched. That’s an acceptable failure mode for a family archive. It might not be for your bank.

The Takeaway

“Duplicate detection” sounds like it wants an index and hashes and a schema. Often it wants: ask the file system if the file’s already there, and keep a note of what you’ve handled. Two humble ideas, no database, and a pile of 10,000 files that I could finally chip away at a little each day without ever doing the same work twice.

Next, back to the fun end of the pipeline: what watching the finished archive actually feels like.