You can use the commands grep and xargs in the following way:
grep -rlZ '=head1 NAME' /target-directory/ | xargs -0 rm -f
The grep command will work recursively -r (within the /target-directory/) and will output only the names of the files -l that contain the pattern '=head1 NAME'. The -Z option will output a zero byte (the ASCII NUL character) instead of the character that normally follows a file name.
This output will be piped | to the stdin of the xargs command where -0 means that the input items will be terminated by a null character instead of by whitespace. The command that will be executed is rm -f that means force remove.
Thanks to @steeldriver for the final improvement of the answer!
In addition you can add -L1 (read one line per execution) to the xargs command and echo to perform dry run:
grep -rlZ '=head1 NAME' /target-directory/ | xargs -L1 -0 echo rm -f
If you want to move the files somewhere instead of delete them, modify the command in this way:
grep -rlZ '=head1 NAME' /tgt-dir/ | xargs -L1 -0 -I {} echo mv {} /destination-dir/
Or as @PerlDuck said "mv also has a switch -t, --target-directory=DIRECTORY which allows to put the directory first and the files last... This way mv isn't called once for each single file but once for a bunch of files.":
grep -rlZ '=head1 NAME' /tgt-dir/ | xargs -0 echo mv -t /destination-dir/