I use Gnome Tweaks Tool 3.34.0 (and GNOME Shell 3.36.4) and under "Keyboard & Mouse" I change via "Additional Layout Options" the "Caps Lock Behavior" to be swapped with Escape key.

Yet while this works for a while, the setting often undoes itself and I don't know why. The setting is seen as toggled, but the default behavior of the button is restored.
How can I make the swap permanently?
The issue occurs often when I lock the screen and login. Also, I have an USB switch to use my input devices at another PC. So it may have to do with the unlpugging the keyboard and then restoring the connection.
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.1 LTS
Release: 20.04
Codename: focal