0

I want to copy a 70GB directory from an exFat external hard drive formatted on a MacOS system to the internal harddrive of a linux Ubuntu LTS 24.04 system. Each time that I tried to transfer the file (either with rsync or cp the entire linux system crashes in the sense that the computer stops responding, the screen is frozen so that I have to hard reboot it by pressing the power button. I did not have problems when transfering other directories from the backup. In the culprit directory, there are approximately 411567 directories and files (using tree subdir > out and counting the line of out. This directory is containing many picture and code (it countains notes and exercices for image recognition deep learning algorithm.)

How is it possible to narrow down the search to the cuplrit file from that 70GB directory?

(This is the follow up of a very long question there: Copying files (cp and rsync) from rom MacOS formatted exFat sytematically crashes entire system (Ubuntu LTS 24.04) which I didn't delete as it contains many step upstream from the current situation). If a moderator thinks that I should delete one of the two question, I would thenk like to delete the previous one and keep the current one

Here is the output of $sudo journalctl -b -1 -ex (however the crash happened at 01:47:34 and I don't see any particular warning signs, do you?)

May 11 01:44:08 mylenovo-ThinkPad-P1-Gen-6 systemd[2739]: launchpadlib-cache-clean.service - Clean up old files in the Lau>
░░ Subject: A start job for unit UNIT has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit UNIT has finished successfully.
░░
░░ The job identifier is 786.
May 11 01:45:01 mylenovo-ThinkPad-P1-Gen-6 CRON[5609]: pam_unix(cron:session): session opened for user root(uid=0) by root>
May 11 01:45:01 mylenovo-ThinkPad-P1-Gen-6 CRON[5610]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
May 11 01:45:01 mylenovo-ThinkPad-P1-Gen-6 CRON[5609]: pam_unix(cron:session): session closed for user root
May 11 01:46:05 mylenovo-ThinkPad-P1-Gen-6 /usr/libexec/gdm-x-session[2892]: (EE) event5  - ELAN0686:00 04F3:320D Touchpad>
May 11 01:46:05 mylenovo-ThinkPad-P1-Gen-6 /usr/libexec/gdm-x-session[2892]: See https://wayland.freedesktop.org/libinput/>
May 11 01:46:06 mylenovo-ThinkPad-P1-Gen-6 /usr/libexec/gdm-x-session[2892]: (EE) event5  - ELAN0686:00 04F3:320D Touchpad>
May 11 01:46:06 mylenovo-ThinkPad-P1-Gen-6 /usr/libexec/gdm-x-session[2892]: See https://wayland.freedesktop.org/libinput/>
May 11 01:46:17 mylenovo-ThinkPad-P1-Gen-6 /usr/libexec/gdm-x-session[2892]: (EE) event5  - ELAN0686:00 04F3:320D Touchpad>
May 11 01:46:17 mylenovo-ThinkPad-P1-Gen-6 /usr/libexec/gdm-x-session[2892]: See https://wayland.freedesktop.org/libinput/>
May 11 01:46:17 mylenovo-ThinkPad-P1-Gen-6 /usr/libexec/gdm-x-session[2892]: (EE) event5  - ELAN0686:00 04F3:320D Touchpad>
May 11 01:46:17 mylenovo-ThinkPad-P1-Gen-6 /usr/libexec/gdm-x-session[2892]: See https://wayland.freedesktop.org/libinput/>
May 11 01:46:35 mylenovo-ThinkPad-P1-Gen-6 /usr/libexec/gdm-x-session[2892]: (EE) event5  - ELAN0686:00 04F3:320D Touchpad>
May 11 01:46:35 mylenovo-ThinkPad-P1-Gen-6 /usr/libexec/gdm-x-session[2892]: See https://wayland.freedesktop.org/libinput/>
May 11 01:46:35 mylenovo-ThinkPad-P1-Gen-6 /usr/libexec/gdm-x-session[2892]: (EE) event5  - ELAN0686:00 04F3:320D Touchpad>

There was no problem of memory as shown with the free a few seconds before the crash:

               total        used        free      shared  buff/cache   available
Mem:        32515972     3066244    19160136      730976    11487144    29449728
Swap:        8388604           0     8388604

EDITS

  • Maneuver 1

I ran the command tar cvf /dev/null /medial/mylenovo/bckp 2>&1 > temp_tarout. The process ended without any crash and there was no error that I could find in tarout.

  • Maneuver 2

I ran the following command:

# name of the file: script1
#!/bin/zsh
dest="$HOME/blah/blah"
src="/path/to/90GB/dir"

find $src -type f -print0 | 

  xargs -0 -r cp -t  $dest

saved the output into the file with zsh script1 >out1 and it crash during the run of the command

# name of the file: script2
#!/bin/zsh
dest="$HOME/blah/blah"
src="/path/to/90GB/dir"

find $src -type f -print0 | 

  xargs -0 -r cp -t --no-clobber $dest

saved the output into the file with zsh script1 >out2 but got the following error message:

cp: target directory '--no-clobber': No such file or directory
cp: target directory '--no-clobber': No such file or directory
cp: target directory '--no-clobber': No such file or directory
cp: target directory '--no-clobber': No such file or directory
  • Maneuver 3

I copied with rsync the incriminated directory from one hard drive to another hard drive, connected to the computer with Ubuntu 24.04. The problem persisted: it crashed again

  • Maneuver 4 Disabling the swap with the following command did not solve the problem:
sudo swapoff -a
ecjb
  • 225

3 Answers3

3

Your system shouldn't crash. After your system comes up, use sudo journalctl -b -1 -ex to look at the messages just before the crash.

Do you have enough swap space to avoid the OOM-killer? free will show.

A solution:

#!/bin/bash
dest="$HOME/blah/blah"
src="/path/to/90GB/dir"

find $src -type f -print0 | 

  xargs -0 -r cp -t --no-clobber $dest

This will copy the files one at at time, add the -v option to cp to see each copy.I later added the --no-clobber, on case it was run multiple times. -

waltinator
  • 37,856
0

I had the same problem when upgraded to Ubuntu 24.04 on a new machine. Transferring data from an external drive (200GB+) kept freezing the system to the point I had to do a hard reset (mouse not moving, keyboard not responding). I noticed it to be an OOM error, but I'm on a 64GB ram machine. This shouldn't be happening.

Tried increasing and decreasing swap and vm.min_free_kbytes

Eventually disabling the swap is what fixed it:

sudo swapoff -a

candy_before_dinner
  • 1
-1

Switching from Ubuntu LTS 24.04 to 22.04 solved the problem (without being able to explain in more details why)

ecjb
  • 225