0

I got say 4TB of millions of files. I think if I can produce a single .md5 for each file, it would be nice.

I read Generate a separate .md5 file for each .fastq file in a directory? and modified to make it produce an individual .md5 for EACH individual file.

say : find . -type f -name "*" -exec sh -c "md5sum {} > {}.md5" \;

But as I say I got 4TB of millions of files, the command may not finish in 2 days and I have to shutdown the PC.

when resume, I wish the above command could skip those files that already have existing .md5 files (i.e. those calculated).

How to do this? recursively, of course.

thank you.

charles chan
  • 1

1 Answers1

1

Rather than create one .md5 file per file ("millions of files"), using up millions of disk clusters and millions of inodes, put .md5 signatures for all files in one big file (which you can split up later, if you wish)(read man find;man xargs): 

find . -type f \! -name '*.md5' -print0 | xargs -0 md5sum > md5sum.out
#   Separate filenames with \000^^^^^^^          ^ in case some have spaces
#              ^^^^^^^^^^^^^^^^ but NOT *.md5 files
#      ^^^^^^^ each file
waltinator
  • 37,856