Decrease of transfer rate when copying large amount of data
Decrease of transfer rate when copying large amount of data
Very slow copying files from ExFat to NTFS
Hardware Forum 11/22
https://ubuntuforums.org/showthread.php?t=2502523&page=2
The "slow copy" or "decrease of transfer rate when copying large amount of data" are known problems, with many askubuntu questions and launchpad bugs filed. Previous postings have covered most of the things to check, but to
reiterate:
1 ) Check the specs on Your MB PCIE M.2 slots -- if different, use a fast one (PCIE5 vs PCIE4) for your target.
2)Ensure your partitions/filesystems are aligned properly -- 1MB or even 4MB.
3)System buffers fill up when writes are slower than reads, which is the usual case. Things slow down when buffers fill. Using the nocache command on a terminal copy will at least prevent the input stream from being buffered.
4)The "time" command is usless for determining write speeds, the copying process finishes minutes before the system buffers get flushed. Force the flush with a sync or watch the blinking lights for disk activity to determine the actual finish of the copy.
5)Some copy commands (tar, cpio,...)allow you to specify a buffer size -- this may help in limiting memory fragmentation, speeding things up a bit. The tar
command -L size option will pause for user input after "size" kiB have been written.
Separately monitor the available memory (free command), and only continue when enough available memory is present (disk buffers will be flushing).
6)There are tweaks that may help for specific cases -- different scheduler, altering dirty write bits or rations, but the filling of system buffers will eventually drag things to a crawl. Slowing the reads down to the write speeds is the goal of such alterations. Most of the suggestions for tweaks were for hard disks, SSDs are likely to have completely different optimizations.
Follow the previous suggestions for seeing what helps your situation the most.
Understand, this is an unresolved problem that prevents some people from using Linux for their work -- they simply cannot back up their data sets in a timely manner. Good luck, and interested in seeing what you find helps you the most.
For some insight into the problem, the free command will show you swap space used, and the /proc/buddyinfo file has memory fragmentation information. The alt+sysreq (or echo m > /proc/sysrq-trigger) should cause memory info to be entered into /var/log/messages.
On a 16GB memory, 11gen I7 11800H cpu, USB3.2 gen1 with a two TB SSD as target, a 300 GB directory copy
from an internal SSD (PCI4)took approx 30 min with the below command:
sudo nocache tar c --record-size=500M -f- . |(cd /mnt/a2; sudo tar -xpBf -)
No noticeable impact on system performance was noticed (cursor never froze etc.).
