How do I fix CUDA breaking after suspend?

Question

So as the title says, everytime I wake my laptop up from a suspend, CUDA breaks, just plain doesn't work. I have to reboot my system in order to get it back. So to get it back after a suspend I have to run 'sudo rmmod nvidia_uvm' and then 'sudo modprobe nvidia_uvm' and of course that works, but as you'd imagine that gets old real quick. So how do I get it back permanently? If it makes any difference, I have a PRIME laptop with my Nvidia GPU always selected (940MX) with the 440 drivers.

Answer 1

It looks like the nvidia_uvm module is not working well with suspending. CUDA can be used again if the module is reloaded:

sudo rmmod nvidia_uvm
sudo modprobe nvidia_uvm

Unfortunately this does not work when there are CUDA applications active.

I have filed a bug report: https://bugs.launchpad.net/ubuntu/+source/nvidia-graphics-drivers-460/+bug/1921477

Answer 2

I had the same problem on ubuntu. Recently switched to Archlinux. Followed this tip on the wiki and it has fixed this problem on my gtx1650 laptop.

This way one can activate a new power management tool. See nvidia documentation.

My steps:

1. Create /etc/modprobe.d/nvidia-power-management.conf with

options nvidia NVreg_PreserveVideoMemoryAllocations=1 NVreg_TemporaryFilePath=/tmp

(on one line in the file).

2. Update initramfs. On ubuntu that is via update-initramfs -u
3. systemctl enable nvidia-suspend.service. I think you have to manualy install the systemd service files first on Ubuntu. See this post
4. Reboot

Answer 3

I have the same issue trying to use Ollama.

I am running this script every time my laptop return from suspend...:

1. Stop ollama
2. Call daemon-reload
3. Remove nvidia_uvm
4. Load nvidia_uvm
5. Start ollama

The script:

#!/bin/bash
echo "Stopping ollama"
systemctl stop ollama
echo "Calling daemon reload"
systemctl daemon-reload
echo "Removing nvidia_uvm"
rmmod nvidia_uvm
echo "Loading nvidia_uvm"
modprobe nvidia_uvm
echo "Starting ollama again"
systemctl start ollama

Answer 4

This problem is closely linked to a question I opened GPU not found errors.

nvidia-driver-550 with cuda-12.4 works with cccplex's solution. But there's no need to enable the suspend service in the newest drivers.

What you would want to copy and paste in the file /etc/modprobe.d/nvidia-power-management.conf is

options nvidia NVreg_PreserveVideoMemoryAllocations=1
options nvidia NVreg_TemporaryFilePath=/tmp

If you try to enable the suspend and resume services, you'll find that it's masked, DO NOT unmask it or else it gets removed. As I suppose one might expect, the service is only utilized on suspend and resume.

Answer 5

I do not have enough points to add a comment to cccplex answer but it works aswell with ubuntu 24.04 and geforce 4090 with driver 550