47

Ubuntu 22.04 comes with the systemd-oomd service enabled by default, which has been "helpfully" killing my IDE and / or terminals whenever I try to compile an application using an abundance of threads / memory.

What is the right way to either turn this off, or configure the service to not shoot random processes in the face while I'm using them?

I'm aware that I can mitigate this behavior in a few ways; e.g. by increasing the size of the swap space, but this is still not a panacea since:

  • The OOM daemon kills the entire process tree, so even the terminal hosting the processes that were killed will suddenly vanish;

  • The OOM daemon kills the process tree without providing any notification to the user, so all the user knows is that their terminal / IDE / application hosting memory-hungry processes has suddenly vanished.

A user could find out what happened post-hoc via journalctl or something similar if they knew what to look for, but I don't think the average Ubuntu desktop user would think to do this.

As an example, normally when a process crashes via a deadly signal or similar, a crash reporter will tell the user that something went wrong. Shouldn't there be a similar facility for processes killed by the OOM daemon?

Edited to add requested output re: swap space; as far as I know these are just the defaults that were set when Ubuntu 22.04 was installed.

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            31Gi       5.2Gi       3.1Gi       210Mi        23Gi        25Gi
Swap:          2.0Gi       0.0Ki       2.0Gi

$ sysctl vm.swappiness
vm.swappiness = 60


$ swapon -s
Filename                Type        Size        Used        Priority
/swapfile                               file        2097148     792     -2
Kevin Ushey
  • 951

5 Answers5

38

Most systemd services can be managed via the systemctl utility. In this case, we want to disable the systemd-oomd service. This can be done with:

$ systemctl disable --now systemd-oomd

You should see something like (depending on your OS):

$ systemctl disable --now systemd-oomd
Removed /etc/systemd/system/multi-user.target.wants/systemd-oomd.service.
Removed /etc/systemd/system/dbus-org.freedesktop.oom1.service.

You can then verify that the service is disabled, with:

$ systemctl is-enabled systemd-oomd

And you should then see:

$ systemctl is-enabled systemd-oomd
disabled

It is possible, however, that other services might attempt to restart the systemd-oomd service. To prevent this, you can 'mask' the service. For example:

$ systemctl mask systemd-oomd
Created symlink /etc/systemd/system/systemd-oomd.service → /dev/null.

And then systemctl is-enabled should now report:

$ systemctl is-enabled systemd-oomd
masked

If you'd like to later unmask (or re-enable) a service, that can be done with, for example,

$ systemctl enable systemd-oomd
$ systemctl unmask systemd-oomd

See man systemctl for more details; in particular, note the caveats regarding masking of systemd services.

Kevin Ushey
  • 951
12

Just to point out, systemd-oomd doesn't just kill on oom (out of memory and swap) -- if it did, that'd probably be fine. It (either also or exclusively, not sure which..) kills based on what it deems excessive swap activity (the message I found on mine indicates Killed (gnome-terminal's scope) "due to memory pressure for " (the slice) "being 58.97% > 50.00% for >20s with reclaim activity."

I (this is the same in Ubuntu 20.04 and 22.04) run some junk all day while I'm away, it gradually swaps out bits of the copy of firefox that's open, when I get home, go to use firefox at which point it has to swap some of it back in. Which is fine, but system-oomd (this has happened to me twice within a few days after upgrade to Ubuntu 22.04) decided it's not OK and killed my gnome-terminal with a screen session with like 8 or 9 shells, VirtualBox, some s3ql mounts, etc. all running under it. I was really afraid something in my upgrade had gone wrong and gnome-terminal and such were crashing on their own, glad to see that's not the case. Thanks systemd! Anyway, at least it's easy to disable!

Edit: FYI, I'm able to run:

sudo apt remove systemd-oomd

as well. It's apparently just installed directly, not as a dependency of the systemd package or anything like that.

andrew.46
  • 39,359
hwertz
  • 376
4

Bit of a necro. You can increase swap size but, changing swap size to 4gb from default 2gb does nothing for me. oomd.conf can be edited so you can try that. My drive is small so I don't want to increase swap size any further than 8gb and I don't want to turn swap off either. As far as I know, oomd is very important for not crashing your whole pc and corrupting everything from running out of memory. So I don't want it turned off completely.

systemd-analyze cat-config systemd/oomd.conf

https://man.archlinux.org/man/oomd.conf.5.en

https://man.archlinux.org/man/systemd-oomd.service.8.en

Also I have no idea what I'm doing so do this at your own risk. haha. It's read-only for a reason I guess.

Since it is read-only you can make a file and then edit it by doing this

sudo touch [filename]
sudo nano [filename]

I've changed duration to 60s and pressure to 90%. Also swap limit to 95%. Also for testing you can add the flag --dry-run to systemctl stuff. Instead of doing what it would do it prints what it would do to a log.

sudo systemctl --dry-run start systemd-oomd

EDIT:

new solution: delete this trash

sudo apt remove systemd-oomd
Kermit
  • 41
3

With 32G RAM, the default 2G /swapfile would normally be enough. However if you're running some large memory-hungry apps, or have large data-sets, it may need to swap more, so we'll increase it from 2G to 4G and see if that helps. Don't disable OOM.

Note: Incorrect use of the rm and dd commands can cause data loss. Suggest copy/paste.

In the terminal...

sudo swapoff -a           # turn off swap
sudo rm -i /swapfile      # remove old /swapfile

sudo dd if=/dev/zero of=/swapfile bs=1M count=4096


sudo chmod 600 /swapfile  # set proper file protections
sudo mkswap /swapfile     # init /swapfile
sudo swapon /swapfile     # turn on swap
free -h                   # confirm 32G RAM and 4G swap

Edit /etc/fstab, using sudo -H gedit /etc/fstab or sudo pico /etc/fstab.

Confirm this /swapfile line in /etc/fstab... and confirm no other “swap” lines... use SPACES in this line... confirm NO TABS...

/swapfile  none  swap  sw  0  0

reboot                    # reboot and verify operation
heynnema
  • 73,649
0

I had a very similar problem as discribed by hwertz. Firefox caused a crash of the whole system.

Since neither increasing the swap nor disabling/masking the systemd-oomd service did the job on my system, I want to add my solution here.

In my case it is a rather old system with an NVIDIA Geforce 9500 GT. The card requires the 340 driver which is not included anymore in Ubuntu 22.04. Already after the upgrade from 20.04 I had problems (no login screen,..) and I had to remove all NVIDIA* packages via

sudo apt purge nvidia*

Well using nouveau solved the login problem, but triggered the Firefox freeze. Finally the problem's solution on my old system was to get a working 340-driver version for Ubuntu 22.04.

Note: I am not sure if this would also happen with a fresh installation. Can be that there is a misconfiguration of the system due to the upgrade from 20.04

aaigner
  • 1