Moving large amount of files (~ 100 000)

Question

I work with folders that contain a lot of files, like 100 000 or even 1 000 000 files per folder. When I try to move the content of a folder into another, my computer always gets stuck. Even when the process seems finished, I can't see the content of any folder because nautilus seems completely frozen and I have to force my computer to restart. I noticed that this happens also when I try to move like 10 000 files.

Is that a problem of my computer or is it normal when working with these numbers?

Any smart way of performing this file transfer?

Answer 1

Perhaps consider using a pure command line method to transfer very large amounts files, you will undoubtedly find the process is substantially faster than using a gui.

There are many different ways to accomplish this, but the following worked quickly, safely and efficiently on my system:

find . -maxdepth 1 -type f -print0 | xargs -0 mv -t <destination>

Some explanation for this command:

1. Your input directory is the '.' character and for this particular command you need to be in that directory
2. Your output directory is the <destination> in my example. Obviously modify this to suit your own needs and leave out the brackets.
3. This syntax allows for filenames with spaces as a bonus :)

Endless permutations are possible but this should work well and much more efficiently than the gui. One permutation for example: if you wanted to move only pdf files you could run:

find . -iname "*.pdf" -maxdepth 1 -type f -print0 | xargs -0 mv -t <destination>

Use of xargs opens many possibilities particularly with the movement of such a large number of files. Many, many possibilities....

Potential Problems:

There are at least 2 potential pitfalls to ponder, thanks to the commenters below for these thoughts:

1. Your destination directory could be corrupt, in a subsequently unreachable location, mistyped etc. mv will still move the files there! Be careful here...
2. If the -t option (--target-directory) is missing and the destination folder is actually a file you will move one file and fail on the rest. mv has 2 uses: rename source to destination or move source to directory. Again be careful...

Answer 2

I had similar experience before, it is normal when dealing with a large number of files. I was having a large collection of PDF data-sheets (electronic parts).

GUI tools check for some file details & meta data (Icon/Thumbnail, Size, ...), it will be a big deal in such case. Even in Icon View and without thumbnails, they will freeze as most of them are not designed for such extreme case. GUI tool try to load presentation icons for all files/folders in directory even those items are not visible to user in current screen portion. Sorting is also part of the problem and no way to avoid it.

• I end up splitting files files on separate folders based on brand/model less then 10000 each. May be you can use date (as most people do with photos/scans) or first letter(s) (as in Ubuntu package repository)
• It is easier to use CLI tools instead as they show only what you have requested. You can use locate for quick search in stead of find.

• For move operation, use mv in terminal (GUI tools are slow because they try to update the view periodically).

  If it is in the same partition, the command will change only the pointers in the File-system index. If not, then it will be dual operation (copy & delete). That will be expensive.

There is only one case I can help, If you are copying those files multiple times and they are not updated. As I did when I share my collection with friends, each time I try to copy it takes a decade. (This is more useful with small size files only)

• Create a single package or few packages, like zip with no/low compression. When you copy it, it will faster, so let DMA do its job.

Answer 3

If you are looking for a solution which gives you the benefits of the command-line operations with the GUI feeling and flexibility combined, I recommend mc (midnight commander).

It is a ncurses-based visual file manager - you have a two-pane view on your files and a menu available. Use of the mouse is possible, even over ssh. You can browse around your fs, inspect files with the file viewer, filter according to criteria on-the-fly and have the copy or move operations done on the command line.

It's a clone of the DOS program Norton commander which was popular in the mid-Eighties. It works well whenever the GUI starts to get unreliable for me, and ideal for your purpose.

Answer 4

I've encountered somewhat similar problems - I was testing my RAID setup and when doing huge transfers (e.g. 100,000+ files and 1-2 TB of data in one go) it seems like transfers start fairly speedy - lets say ~200MB/sec, then quickly slow down to a reasonable plateau ~90-120MB/sec (possibly after consuming some flash cache storage on the drives). Then after 20-30min, the operation gradually starts to drop to a much lower plateau ~30-40MB/sec, worse when dealing with small files - taking a 4-5 hour operation closer to 15hours.

I spent some time trying to diagnose - e.g. possible drive faults. Despite trying different tools - command line, nautilus, I couldn't maintain a decent throughput for very large copy operations.

What worked best for me was to use midnight commander, and whenever copying got slow, I would pause the operation until the hard-drive light went out after any pending operations flushed out - usually a minute or so - then un-pause MC again and it would shoot back up to a decent pace for another 20-30min. Rather annoying though.