How to delete duplicated files based on pattern (see examples below) using a script or a Bash command

Question

My question is simple :

I have video files named as follows :

• xxxx_yyy_720_3800.mp4
• xxxx_yyy_720_8000.mp4

where yyy can vary in length (e.g yyyyyy or more y)

As I'm not used in shell scripts I'd like to delete automatically (in a specified folder)

• If a file with the name xxx_yyy_720_3800.mp4 exists and if the file xxx_yyy_720_8000.mp4 exists, delete xxx_yyy_720_3800.mp4
• If there is only a file xxx_yyy_720_8000.mp4 and not a file xxx_yyy_720_3800.mp4, do nothing.

Any help would be greatly appreciated.

Answer 1

Using find and gawk

1. Install gawk

   sudo apt-get install gawk
   

2. Go into your folder or replace the . after the find command with your foldername, eg: find ~/my_video_duplicates f -iname …

3. Test the command

   The command below shows only the remove candidates

   find . -type f -iname "*_8000.mp4" -print0 | \
       while read -d $'\0' file; do \
           gawk -F_ '{ \
               a=gensub(/\_8000\./, "_3800.", "g" , $0); \
               system("if [ -f \""a"\" ]; then echo \""a"\" will be deleted; fi")}' <<< "$file";\
       done
   

4. Check again, if you are in the right folder or replace the . after the find command with your foldername, eg: find ~/my_video_duplicates f -iname …

5. If you sure, run the command below

   find . -type f -iname "*_8000.mp4" -print0 | \
       while read -d $'\0' file; do \
           gawk -F_ '{ \
               a=gensub(/\_8000\./, "_3800.", "g" , $0); \
               system("if [ -f \""a"\" ]; then rm \""a"\"; fi")}' <<< "$file";\
       done
   

---

Example

• The starting situation

  % ls -og
  total 3
  -rw-rw-r-- 1 0 Jul 14 19:37 xxxx_yyy_720_3800.mp4
  -rw-rw-r-- 1 0 Jul 14 19:20 xxxx_yyy_720_8000.mp4
  -rw-rw-r-- 1 0 Jul 14 19:21 aaaa_yyy_720_8000.mp4
  

• The dry run

  % find . -type f -iname "*_8000.mp4" -print0 | \
      while read -d $'\0' file; do \
          gawk -F_ '{ \
              a=gensub(/\_8000\./, "_3800.", "g" , $0); \
              system("if [ -f \""a"\" ]; then echo \""a"\" will be deleted; fi")}' <<< "$file";\
      done
  ./xxxx_yyy_720_3800.mp4 will be deleted
  

• The removal

  % find . -type f -iname "*_8000.mp4" -print0 | \
      while read -d $'\0' file; do \
          gawk -F_ '{ \
              a=gensub(/\_8000\./, "_3800.", "g" , $0); \
              system("if [ -f \""a"\" ]; then rm \""a"\"; fi")}' <<< "$file";\
      done
  

• The final situation

  % ls -og
  total 2
  -rw-rw-r-- 1 0 Jul 14 19:20 xxxx_yyy_720_8000.mp4
  -rw-rw-r-- 1 0 Jul 14 19:21 aaaa_yyy_720_8000.mp4
  

Answer 2

Using the Bash Shell

[ -f "file" ] checks whether a filename exists, and is an ordinary file (e.g. not a directory or symbolic link)

"${name/%x/y}" replaces the suffix x of $name with y.

Therefore to delete xxx_yyy_720_3800.mp4 only if xxx_yyy_720_8000.mp4 exists, where xxx and yyy are identical in each case, for all *_*_720_3800.mp4 in the current directory:

for name in *_*_720_3800.mp4
do if [ -f "${name/%3800.mp4/8000.mp4}" ]
   then echo "$name"
   fi
done

Change the echo to rm if you are sure this is OK.