I am doing machine learning on an Nvidia 2080TI GPU using PyTorch. It has 11GB of DRAM and I have 32GB of CPU RAM. I have to carefully tune my batch to fit inside 11GB. Using CUDA unified memory I should be able to prevent CUDA out of memory errors by extending the DRAM space into RAM. Is this possible on Ubuntu, and if so, how do I configure it?
I'm asking because GPU price goes up exponentially with the size of DRAM, regardless of how old the GPU is, but RAM is comparatively cheap. I'm trying to cover boundary cases where most of the time things run in the 11GB and occasionally there is an overage I need to cover.
UPDATE: It seems PyTorch doesn't use this mechanism. However, a separate package called SpeedTorch makes it available in practice. However it requires some fairly deep knowledge of the neural net architecture and the CPU to apply it effectively.
