I am in the process of switching from MacOS (MacBook Pro, Retina, 15-inch, Mid 2015, 16GB RAM) to Linux. I installed Ubuntu 24.04 LTS about 10 days ago. Copying files (using both rsync and cp) from MacOS formatted exFat systematically crashes entire system (Ubuntu LTS 24.04) 1-2 minutes after launching to the point where I had to hard reboot the computer by pressing the power button. I have reasons to believe that tracker or tracker-miner-f could be the culprit as this was the last command executed (with an error ) just before the crash documented in /var/log/syslog
What has PROBABLY been excluded so far:
-
tracker-miner-f: crash persisted after removingtracker
-
- It seems to not be a hardware hard drive related problem as the crash occured while transferring same data from 2 different hard drives as source (both formated in exFat).
-
- It seems not to be an
rsyncspecific issue as usingcpinstead ofrsyncto transfer data also crashed
- It seems not to be an
-
- It seems not to be a RAM issue (as the RAM and Swap usage was relatively low on crash)
-
- RAM problem on source (MacOS): MemTest successfully run: did not detect any error
What could it be ??:
- exFat related formatting problem?
- corrupt file
Next steps (being done or to be done):
- eleminate the corrupt file
- ANY OTHER IDEAS (Please...)???
System
Here are some data regarding my system:
OS: Ubuntu 24.04 LTS x86_64
Host: ThinkPad P1 Gen 6
Kernel: 6.8.0-31-generic
Shell: zsh 5.9
Resolution: 2560x1440
DE: GNOME 46.0
Terminal: tmux
CPU: 13th Gen Intel i7-13700H (20) @ 4.800GHz
GPU: Intel Raptor Lake-P [UHD Graphics]
GPU: NVIDIA RTX 2000 Ada Generation Laptop GPU
Memory: 4211MiB / 31753MiB
Data to be transferred
The data flow is the following I saved with rsync 800GB of data from MacOS (MacBook Pro, Retina, 15-inch, Mid 2015) to an external hard drive formated in exFat on the exFat. I then tried to save back a subset of the data (70 GB) from the external hard drive to the Linux/Lenovo/Ubuntu 24.04 system described above with the rsync 3.2.7
Here are some data about the 70GB directory that I want to backup. It has approximately 411567 directories and files (using tree subdir > out and counting the line of out. This directory is containing many picture and code (it countains notes and exercices for image recognition deep learning algorithm.)
DETAILED DESCRIPTION OF SEVERAL EXCLUDED CAUSE
1. Probably not tracker related because
Here is a copy of the last 5 minutes before a crash of /var/log/syslog:
2024-05-10T12:54:54.572824+02:00 mylenovo-ThinkPad-P1-Gen-6 tracker-miner-f[3698]: message repeated 8 times: [ Could not execute sparql: Constraint would be broken: UNIQUE constraint failed: rdfs:Resource.ID]
2024-05-10T12:55:01.399803+02:00 mylenovo-ThinkPad-P1-Gen-6 CRON[5519]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
2024-05-10T12:55:03.177883+02:00 mylenovo-ThinkPad-P1-Gen-6 tracker-miner-f[3698]: Could not execute sparql: Constraint would be broken: UNIQUE constraint failed: rdfs:Resource.ID
2024-05-10T12:55:11.905705+02:00 mylenovo-ThinkPad-P1-Gen-6 tracker-miner-f[3698]: Could not execute sparql: Constraint would be broken: UNIQUE constraint failed: rdfs:Resource.ID
2024-05-10T12:55:19.189509+02:00 mylenovo-ThinkPad-P1-Gen-6 rtkit-daemon[1898]: Supervising 9 threads of 6 processes of 1 users.
2024-05-10T12:55:19.189597+02:00 mylenovo-ThinkPad-P1-Gen-6 rtkit-daemon[1898]: Supervising 9 threads of 6 processes of 1 users.
2024-05-10T12:55:20.730061+02:00 mylenovo-ThinkPad-P1-Gen-6 tracker-miner-f[3698]: Could not execute sparql: Constraint would be broken: UNIQUE constraint failed: rdfs:Resource.ID
what I can see maybe of interest is that there are several instances of the following error:
[timestap] mylenovo-ThinkPad-P1-Gen-6 tracker-miner-f[3778]: Could not execute sparql: Constraint would be broken: UNIQUE constraint failed: rdfs:Resource.ID
This is also the last message (with error) recorded on at least 3 different crashes.
I saw the following question related to tracker-miner
tracker-store and tracker-miner-fs eating up my CPU on every startup and tried the following command:
systemctl --user mask tracker-store.service tracker-miner-fs.service tracker-miner-rss.service tracker-extract.service tracker-miner-apps.service tracker-writeback.service
tracker reset --hard
but got zsh: command not found: tracker. So I tried
tracker3 reset -s
tracker3 reset -r
tracker3 reset -f
and rebooted, rerun a copy script, still crashed but tracker was still on in /var/sys/syslog
Finally I removed tracker with
sudo apt-get remove --purge tracker
the crashes persisted
2. Probably not hardware / Hard drive related because: I made 2 backups of file on 2 different hard-drives but the crash occur on both hard-drives. So it seems not to be related to hard-drive
3. Probably not cp/rsync related because:
Here is the first script I used script.zsh (with same flags copied from my MacOS machine):
#!/bin/zsh
set -o errexit
set -o nounset
set -o pipefail
VOLUME_NAME="bckp_11"
SUBDIR="subdir"
SOURCEDIR="/media/mycomputer/${VOLUME_NAME}/path/to/Documents/${SUBDIR}"
TARGETDIRCURRENT="/home/mycomputer/Documents/macos_documents/${SUBDIR}"
rsync -vaP $SOURCEDIR $TARGETDIRCURRENT --delete || exit 1
Following comments changed the rsync command several times by changing/removing flags. Here are the following command which I ran which also crashed:
This one crashed:
rsync -vvvvv -a --progress --delete "${SOURCEDIR}" "${TARGETDIRCURRENT}" || exit 1
This one also crashed:
rsync -rptgo --progress --delete "${SOURCEDIR}" "${TARGETDIRCURRENT}" || exit 1
This one also crashed:
setopt +o nullglob; for source in "${SOURCEDIR}"/{,.}*; do rsync -a --progress --delete "${source}" "${TARGETDIRCURRENT}"; done
This one also crashed (where the --delete option was deleted) :
setopt +o nullglob; for source in "${SOURCEDIR}"/{,.}*; do rsync -a --progress "${source}" "${TARGETDIRCURRENT}"; done
I ran sudo apt-get update and sudo apt-get upgrade and re-run the script setopt +o nullglob; for source in "${SOURCEDIR}"/{,.}*; do rsync -a --progress "${source}" "${TARGETDIRCURRENT}"; done but it still crashed again
I also tried cp instead of sync with the following script, but it also crashed so it does not seem related to rsync
#!/bin/zsh
set -o errexit
set -o nounset
set -o pipefail
VOLUME_NAME="bckp_10"
SUBDIR="ML"
SOURCEDIR="/media/mylenovo/${VOLUME_NAME}/daily_bckp/Users/mymac/Documents/${SUBDIR}"
TARGETDIRCURRENT="/home/mylenovo/Documents/macos_documents/${SUBDIR}"
cp -r $SOURCEDIR $TARGETDIRCURRENT || exit 1
4. Probably not RAM related on the Linux system because:
I re-ran the commands with htop: when it crashed the MEM bar was visually half full with last bars in yellow, but it said 3.10G/31.06 on the side. The Swp bar was empty with 0K/8.00G (I have 8GB of swap). I conclude that this is not a memory issue, if I am not mistaken (please tell me if you think otherwise)
Some other information in the tmux top pane was the following:
MiB Mem : 31753.9 total, 7666.2 free, 3451.9 used, 21720.0 buff/cache
MiB Swap: 8192.0 total, 8191.7 free, 0.2 used, 28302.0 avail Mem
VIRT RES SHR %CPU %MEM COMMAND
989432 178452 20400 109.3 0.5 tracker-miner-f
2359116 75988 11680 50.2 0.2 DisplayLinkMana
19960 5616 3360 16.3 0.0 tmux: server
981060 98276 52084 10.3 0.3 alacritty
18516 8960 6080 9.3 0.0 rsync
5930600 325752 133912 4.7 1.0 gnome-shell
23108 7116 1280 4.3 0.0 rsync
27.3g 154892 103332 3.7 0.5 Xorg
5. Probably not RAM related on the MacOS system because:
I also ran an memtest (PassMark MemTest86 v10.7 https://www.memtest86.com/) on the MacOS system without problem
