1

My specific situation is that I'm training neural networks on a system which I would also like to use for other work. During the training of some models my PC sometimes doesn't respond for a few seconds, the frame-rate of even relatively light programs like my web-browser becomes unbearably low, and this is to the point where the PC is hardly usable.

This used to not be much of a problem as I was working with small models which would train in 10 minutes during which I could just take a break. But now I'm running into the problem that I have models which I need to train for multiple days, during which my PC is practically unusable.

The desired behavior would be that my 'normal' processes such as using my IDE, text editor or browser get full priority, while the task of training the neural net just gets the computing time that's left.

Is this at all possible in Ubuntu (I'm using 20.04 LTS in case that matters), and if so, how?

Lara
  • 111

2 Answers2

2

There are three options I can think of off the top of my head:

  1. nice ⇢ to run a program with a modified scheduling priority
  2. renice ⇢ to alter the priority of an already-running process
  3. cputool ⇢ to limit the amount of CPU usage a process can access

Option 1 — nice

This is a simple tool that comes built-in with just about every version of Linux that has shipped since the 90s, allowing you to set the priority (niceness) of a process. There are 40 values available ranging from -20 (Top priority! Everything else waits!) to 20 (This can wait).

To set the niceness of a process, you can use this syntax:

nice -n niceness-value [command args]

So, for example, if you're going to gzip compress a massive file but it doesn't need to finish ASAP, you can do this:

sudo nice -n 10 gzip -9 really_big_database_export.sql

This will set the niceness to 10, which is pretty far down the stack.

Option 2 — renice

This allows you to change the priority of an already-running process:

renice -n -15 -p 9999

-n is the niceness and -p is the process ID, or PID. You can find the PID of your processes by using top, htop, glances, and/or ps -eo pid,ppid,ni,comm. There are lots of tools to show this number.

Option 3 — cputool

This one is a bit different from nice and renice in that it allows you to specify the resources available to an already-running process. For example, if you find that your modelling is hitting 100% CPU far too often, you can do something like this:

  1. Use top, stop, glances, and/or ps to get the PID
  2. Call cputool:
cputool --cpu-limit 33 -p 9999

This will limit PID 9999 to use just 33% of the CPU.

If you do not already have cputool on your machine, you can install it with:

sudo apt install cputool

Hope this gives you a little bit of processing power to work with.

1

If you start the process from a terminal you can use nice to start it with a lower priority (see man nice). If you start it from GUI you can use renice to change the priority of the running process.

Be aware that to lower process priority, you need a higher nice/renice value.

But basically, depending on what the bottleneck is (you can use topfor that) the solution normally is more memory (RAM) and/or a faster disk (SSD maybe). Also if the system is low on RAM a faster disk will help on the speed of swapping/paging.

Soren A
  • 7,191