How can I find the memory usage on my GPU?

Question

Previously I was checking the memory usage on my GPU with the following command:

nvidia-settings -q all | grep Memory

I am processing some scientific data on my GPU with numpy and theano. I was doing this with the gnome desktop running, and there was already 380 Mb of memory used on the device. So I stopped lightdm to free up a little more GPU memory; but, now the nvidia-settings tool no longer works.

It simply errors that there is no display

ERROR: Cannot open display ':0.0'.

Is there some other way of getting GPU memory without using this tool? Is there some way to use make it function in headless mode?

Answer 1

For Nvidia GPUs:

nvidia-smi

For Intel GPUs:

intel_gpu_tools

For AMD GPUs:

aticonfig --odgc --odgt

For real time watching -- example:

watch nvidia-smi

Answer 2

I think the better way is to install a utility nvtop

E.g. In the following picture almost 100% memory is used on all 3 GPUs and one gpu processor is used 100%. The other two GPU's are used 0%. command is

nvtop

Answer 3

If you want to check memory usage for every 1 second, then the watch command does the job.

watch -n 1 nvidia-smi

Answer 4

Watch feature is actually built-in to nvidia-smi itself

Run

nvidia-smi -l 1

to monitor memory and other metrics at a 1s interval.