Yesterday I was copying a single, 8 GB file to a USB with a slow, 7 MB/s write speed, while my RAM is 3 GB. While copying the system froze, to the point where I couldn't even move the cursor.
I managed to log into the text console, and ran iotop, it showed that a process named kswapd0 was taking 99.99% of IO.
Are there workarounds so copying a large file doesn't make my system unusable?
According to this bug report I solved it adding following lines
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
into /etc/sysctl.conf
and running
sudo sysctl -p
I ran into the similar issue. Mine is 64 bit Ubuntu 14.04. So After a long struggle I found a answer which solves my issue. For easy use I added the commands below used in that above mentioned answer. Check the answer for detailed explanation.
echo $((16*1024*1024)) > /proc/sys/vm/dirty_background_bytes
echo $((48*1024*1024)) > /proc/sys/vm/dirty_bytes
After using the above command system started to work normally on copying files.
Thanks goes to @Rmano.
I am experiencing a similar issue with the system freezing when copying to a flash drive. I submitted a bug report about it: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1267648
As workaround I found that disabling swap completely eliminates the problem.
I have just had exactly the same problem (in 2019), on ubuntu 19.10, while copying large number of files from USB disk to the SATA disk. Both filesystems are ext4. When I turned off the swap, the problem disappeared. It looks like some bug in memory allocation for disk buffers - apparently, the kernel tries to allocate as much memory for disk buffers, as possible in such situation, which doesn't make sense (making disk buffers in swap...), or it just wrongly calculates the memory size than can be used for caching...
As someone pointed out, setting swappiness to 1 doesn't solve the problem, which is logical, if the files you copy have total size more, than the RAM size...
By the way - can anybody explain, WHY turning off the swap is not recommended? If I have 32GB of RAM, what difference does it make to add another 4GB of swap? I can only think of some obscure applications that actually expect the swap to be there. But I would just stop using such application, since properly written one shouldn't care about swap. The swap should only be managed on the OS level...
Yes, there are kernel settings you can tweak specifying how much data has to be marked as written before it actually gets written to disk. Look here for a pretty comprehensive description of them. In particular, you'll want to find a value of dirty_ratio that works well for you (it's generally too high for desktop/laptop by default, but there's no one magic number that works for everyone).
in Ubuntu 19.10 this problem happens due to problem with swap management . Tinkering with dirty_background broke my nautilus file explorer . Along with freezing when copying large files , i have encountered several performance issues due to bad swap management in 19.10 . Since completely disabling swap is not recommended and we can control the degree of writing into swap partition by setting swappiness .
swappiness=0 tells the kernel to avoid swapping processes out of physical memory for as long as possible . swappiness=100 tells the kernel to aggressively swap processes out of physical memory and move them to swap cache
so , we are going to set swappiness =1 .
To change the swappiness value A temporary change (lost on reboot) with a swappiness value of 1 can be made with
sudo sysctl vm.swappiness=1
To make a change permanent, edit the configuration file with your favorite editor:
sudo gedit /etc/sysctl.conf
Search for vm.swappiness and change its value as desired. If vm.swappiness does not exist, add it to the end of the file like so:
vm.swappiness=1
Save the file and reboot.
This helped me to improve overall performance issues in Ubuntu 19.10 while disabling swap completely eliminated all performance issues that i had with 19.10 but still i won't recommend doing that in normal situations .
As an experiment, I've set sysctl.conf to:
vm.dirty_background_ratio=1
vm.dirty_ratio=2
and this solved the problem. Previous values 5/10 didn't help. Looks like everybody should play with those params to find the best one. Copying of 12 GB file from SSD to pen flash drive took about 15 min without system hangups (USB 2.0). Probably I'll check values 2/4, etc after several days of testing.
System: Ubuntu 19.10, i7, 8GB Ram, SSD.
Turning off swap while copying worked for me.
sudo swapoff -a
After the copying is done, I turn on the swap
sudo swapon
Edit: Do note, as Robby1212 mentions in the comment, that swapping off is dangerous and should be done after you have tried out the other answers. I am fortunate that nothing has happened to me when I swap off. I only swap off during copying large files.
I had similar problems when copying files to an exfat drive. I had less trouble using an ext4 filesystem on my USB hard drive .
In my case as a Linux Mint user, my file transfer issues were caused by the ntfs3 package; disabling this package resolved my issue. altair4's post within this thread has more information, but here's a copy of the original command he/she recommended:
echo 'blacklist ntfs3' | sudo tee /etc/modprobe.d/disable-ntfs3.conf
I then needed to take some additional steps to successfully mount my external drive; see this thread for a list of those steps.
