78,990 Public-Domain Books on My NAS: A Six-Day Download

The Need: a big pile of old books, as plain text, on your own disk

Maybe you want a private source of classic books for your own projects. Maybe you just don’t want your reading to depend on someone else’s website. Either way, the same question comes up: how big is it, and will it fit on my NAS?

For one public-domain library, the answer was 28.1 GiB for 78,990 plain-text files. That fits on a home NAS with room to spare. This post shows the numbers, the commands, and the checks I ran afterwards.

The Proof: what ended up on the disk

Two honest notes about this table:

• My script counts one more .txt file than rsync does (78,991 versus 78,990). I did not track down the difference.
• My first guess was “about 30 GB”. It was a guess, not a measurement. The real number came from the rsync summary at the end.

My NAS is a Synology DS224+. The storage manager showed 1.8 TB free before I started, so space was never the problem. Time was.

The Story: six days, two runs

I did not plan a six-day job. I asked a few questions and then pressed Enter.

Day 1 (Oct 16). I wanted only text, in every language, and I wanted files where non-English characters would not break. I asked:

💬 Prompt that worked “Can’t we include the English ASCII files too?”

That one question changed the plan. Plain ASCII is a subset of UTF-8, so those files are safe. The final filter keeps *-0.txt (UTF-8) and {id}.txt (mostly ASCII) and drops *-8.txt (older ISO-8859 encodings). I also asked whether a dry run was needed, since 1.8 TB was free. I skipped it and started the real download.

The first run had a mistake. It also downloaded the old/ folders, which hold backup copies of earlier editions. I stopped it, deleted those folders, and started again with an exclude rule for old/ placed first.

Day 2 (Oct 17). I moved the script into its own project folder, wrote a CLAUDE.md with the rules, and made the first git commit while the NAS showed 4.1 GB downloaded.

Day 3 (Oct 18). I logged out of SSH. The download died with rsync error code 20. The disk size had stopped at 8,743 MB. The job was started with nohup, and I thought that was enough. It was not. I restarted it with setsid and logged out again. This time the size kept growing. The commit for this fix is stamped 06:32 on Oct 18.

Days 3 to 7 (Oct 18 to 22). Nothing to do but wait. I checked du -sm now and then. The last run reported:

Number of regular files transferred: 53,666
Total transferred file size: 21,063,704,554 bytes
sent 1,970,924 bytes  received 21,077,732,537 bytes  60,856.37 bytes/sec

That is about 60 KB/s. At that speed, 21 GB takes about 96 hours, which matches the clock: the log file was last written at 06:38 on Oct 22. The first run had already brought in about 9 GB, which is why only 53,666 of the 78,990 files were transferred in the second run.

How long was it in total? From about 09:00 on Oct 16 to 06:38 on Oct 22 is roughly 5 days and 21 hours, or about 141 hours. I do not know why the speed was so low. I did not test whether a different time of day or a different mirror would be faster.

The Story, part 2: the cleanup that mostly was not needed

After the download, I wanted to know three things: is everything valid UTF-8, are there duplicates, and is any junk left behind?

1. The check that failed on every file. My first script used iconv to test each file. It reported 78,991 files as not UTF-8, which is every file. The reason: the NAS shell has no iconv command. Each test failed with “command not found”, and my script read that as “bad file”. I rewrote the check in Python (the NAS has python3). The real answer was 706 bad files, not 78,991.

2. Where the 706 files come from. I sorted them into two groups:

• 266 are “covered”. A UTF-8 copy of the same book exists in the same folder, so these are safe to delete.
• 440 are “orphans”. They are the only copy. Their headers say ASCII, Latin-1, or cp1252 (about 329 files), or have no encoding line at all (111 files). None are Korean, Greek, or Cyrillic.

3. The Korean question. Late in the process I asked: “Doesn’t this library have almost no Korean? There is nothing to break.” Right. The real risk is smaller: accented letters and curly quotes (é, ’) in an English book that is stored as Latin-1. So I decided not to convert everything in bulk. When I pick a book to use, I check whether it is in the bad-file list. If it is, I convert only that book.

4. Duplicates. 4,439 books have both {id}.txt and {id}-0.txt. I did not delete any. The rule is simple: if both exist, use -0.txt. With 1.7 TB still free, deleting saves nothing I need, and a delete I regret is worse than a few spare gigabytes.

5. Leftovers. The cleanup script removes any old/ folder that slipped in and empty directories (four passes, because removing one level can empty the one above it).

The How: commands and traps

The download command

setsid nohup rsync -av --stats \
  --exclude='old/' --include='*/' --exclude='*-8.txt' \
  --include='*-0.txt' --include='[0-9]*.txt' --exclude='*' \
  <mirror-host>::<module> "/volume1/<share>/txt/" >> "/volume1/<share>/rsync.log" 2>&1 < /dev/null &

I used the library’s rsync mirror instead of crawling its website, so I did not hit the website at all. These files are public domain.

🗂 Claude.md Rule rsync filter rules are order-sensitive. Keep this order: exclude old/, include */, exclude *-8.txt, include the wanted patterns, exclude * last. Never add --delete: it would remove files you later convert or clean up by hand.

🗂 Claude.md Rule A long job over SSH must be detached with setsid nohup ... >> log 2>&1 < /dev/null &. Plain nohup is not enough on Synology: logging out sent SIGHUP and killed rsync (error code 20).

🗂 Claude.md Rule The Synology shell has no iconv and no pgrep. Check encodings with python3. Check processes with ps aux | grep [r]sync.

The UTF-8 check, in plain terms

The Python script reads every .txt file as bytes and sorts it into three groups:

1. Pure ASCII. Every byte is below 128. Safe.
2. Valid UTF-8. bytes.decode("utf-8") works. Safe.
3. Neither. Written to non_utf8.txt.

It also lists every folder that has both {id}.txt and {id}-0.txt in duplicates.txt. It only reads. It never deletes.

Checking progress without staying logged in

du -sm /volume1/<share>/txt
tail -n 20 /volume1/<share>/rsync.log

I used --stats instead of --progress. With 78,990 files, a progress line per file would make the log huge. The --stats summary prints once at the end.

What I would do differently

• Use setsid from the first run. Skipping that cost me about a day.
• Test the check script on one file before running it on 78,991. The iconv surprise would have shown up in seconds.
• Do a short dry run anyway. It would have shown the old/ folders before I downloaded and deleted them.

Cost

• Software: $0 (rsync, Python, git)
• Download: $0
• Storage: no new disk. About 29 GB of a NAS that had 1.8 TB free.
• Electricity for six days of a NAS that runs all the time anyway: I did not measure it.

What I do not know yet

• Why the transfer speed was so low.
• The one-file difference between rsync’s count and my own.
• Whether the 440 orphan files are really Latin-1 or cp1252. I guess so from their headers, but I have not decoded them all.

The next step is picking a few books and turning them into something I can use. That is a different post.