As stated in the title, my OS automatically schedules almost all work on a single CPU core after upgrading from Ubuntu 20.05 LTS to Ubuntu 22.05 LTS.
I've provided a few screen shots from htop to explain the problem.
In the following example I've run make -j 32 and all make processes are sharing one or two cores rather than using different cores.
1[ 0.0%] 17[ 0.0%] 33[ 0.0%] 49[ 0.0%]
2[ 0.0%] 18[ 0.0%] 34[ 0.0%] 50[ 0.0%]
3[ 0.0%] 19[ 0.0%] 35[ 0.0%] 51[ 0.0%]
4[ 0.0%] 20[ 0.0%] 36[ 0.0%] 52[ 0.0%]
5[ 0.0%] 21[ 0.0%] 37[ 0.0%] 53[ 0.0%]
6[ 0.0%] 22[ 0.0%] 38[ 0.0%] 54[ 0.0%]
7[ 0.0%] 23[ 0.0%] 39[ 0.0%] 55[ 0.0%]
8[ 0.0%] 24[ 0.0%] 40[ 0.0%] 56[ 0.0%]
9[||||||100.0%] 25[ 0.0%] 41[ 0.0%] 57[ 0.0%]
10[ 0.0%] 26[ 0.0%] 42[ 0.0%] 58[ 0.0%]
11[ 0.0%] 27[ 0.0%] 43[ 0.0%] 59[ 0.0%]
12[ 0.0%] 28[ 0.0%] 44[ 0.0%] 60[ 0.0%]
13[ 0.0%] 29[ 0.0%] 45[ 0.0%] 61[ 0.0%]
14[ 0.0%] 30[ 0.0%] 46[ 0.0%] 62[ 0.0%]
15[ 0.0%] 31[||||||100.0%] 47[ 0.0%] 63[ 0.0%]
16[ 0.0%] 32[ 0.0%] 48[ 0.0%] 64[ 0.0%]
Mem[||||||||||||||| 18.0G/62.7G] Tasks: 367, 2174 thr; 34 running
Swp[ 0K/65.0G] Load average: 20.18 7.25 3.32
Uptime: 02:36:39
PID USER PRI NI VIRT RES SHR S CPU%▽MEM% TIME+ Command
22626 samzapo 20 0 85328 76484 4324 R 118. 0.1 0:00.25 htop
21565 samzapo 20 0 2415M 91528 71436 S 19.8 0.1 0:02.10 /snap/firefox/3
22013 samzapo 20 0 396M 313M 17380 R 19.8 0.5 0:01.43 /usr/lib/gcc/x8
22061 samzapo 20 0 289M 260M 17568 R 19.8 0.4 0:01.36 /usr/lib/gcc/x8
22064 samzapo 20 0 392M 304M 17340 R 19.8 0.5 0:01.36 /usr/lib/gcc/x8
22127 samzapo 20 0 375M 293M 16640 R 19.8 0.5 0:01.28 /usr/lib/gcc/x8
It seems that no matter how much work my computer is doing, most of it is going to core 9.
Some work is happening on other cores, but it's as if all processes' affinities are being set to core 9.
I can force the OS to schedule on a different core with, e.g., taskset -c 10-63 make -j 32 and then make will switch to using all cores. But, of course, operating this way is not an option all the time, mainly because other applications all try to use core 9 as well, like my internet browser.
1[ 0.0%] 17[||||||||||||||||||||100.0%] 33[||||||||||||||||||||100.0%] 49[ 0.0%]
2[ 0.0%] 18[||||||||||||||||||||100.0%] 34[||||||||||||||||||||100.0%] 50[ 0.0%]
3[ 0.0%] 19[ 0.0%] 35[||||||||||||||||||||100.0%] 51[||||||||||||||||||||100.0%]
4[ 0.0%] 20[ 0.0%] 36[||||||||||||||||||||100.0%] 52[||||||||||||||||||||100.0%]
5[ 0.0%] 21[||||||||||||||||||||100.0%] 37[||||||||||||||||||||100.0%] 53[ 0.0%]
6[ 0.0%] 22[ 0.0%] 38[||||||||||||||||||||100.0%] 54[||||||||||||||||||||100.0%]
7[ 0.0%] 23[||||||||||||||||||||100.0%] 39[||||||||||||||||||||100.0%] 55[ 0.0%]
8[ 0.0%] 24[||||||||||||||||||||100.0%] 40[||||||||||||||||||||100.0%] 56[ 0.0%]
9[||| 10.0%] 25[ 0.0%] 41[||||||||||||||||||||100.0%] 57[||||||||||||||||||||100.0%]
10[ 0.0%] 26[ 0.0%] 42[ 0.0%] 58[||||||||||||||||||||100.0%]
11[ 0.0%] 27[||||||||||||||||||||100.0%] 43[||||||||||||||||||||100.0%] 59[ 0.0%]
12[||||||||||||||||||||100.0%] 28[||||||||||||||||||||100.0%] 44[||||||||||||||||||||100.0%] 60[||||||||||||||||||||100.0%]
13[||||||||||||||||||||100.0%] 29[||||||||||||||||||||100.0%] 45[ 0.0%] 61[ 0.0%]
14[||||||||||||||||||||100.0%] 30[||||||||||||||||||||100.0%] 46[ 0.0%] 62[ 0.0%]
15[||||||||||||||||||||100.0%] 31[||||||||||||||||||||100.0%] 47[ 0.0%] 63[ 0.0%]
16[|||||||| 27.3%] 32[||||||||||||||||||||100.0%] 48[||||||||||||||||||| 72.7%] 64[ 0.0%]
Mem[|||||||||||||||||||||||||||||||||||||||||||||||48.4G/62.7G] Tasks: 370, 2178 thr; 33 running
Swp[| 31.2M/65.0G] Load average: 21.02 9.82 5.77
Uptime: 02:43:57
PID USER PRI NI VIRT RES SHR S CPU%▽MEM% TIME+ Command
24497 samzapo 20 0 2280M 2222M 22428 R 114. 3.5 0:40.23 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
24509 samzapo 20 0 2088M 2016M 19308 R 114. 3.1 0:39.87 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
24518 samzapo 20 0 2062M 1992M 20404 R 114. 3.1 0:40.21 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
24543 samzapo 20 0 2233M 2173M 22864 R 114. 3.4 0:40.22 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
24546 samzapo 20 0 2508M 2400M 20240 R 114. 3.7 0:40.20 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
24549 samzapo 20 0 2658M 2591M 20324 R 114. 4.0 0:40.22 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
24551 samzapo 20 0 2537M 2469M 19108 R 114. 3.8 0:40.22 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
25145 samzapo 20 0 1609M 1536M 19252 R 114. 2.4 0:24.81 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
25445 samzapo 20 0 1472M 1407M 18832 R 114. 2.2 0:18.84 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
25587 samzapo 20 0 992M 928M 18708 R 114. 1.4 0:15.06 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
25591 samzapo 20 0 1083M 1017M 18708 R 114. 1.6 0:15.01 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
26004 samzapo 20 0 770M 690M 18676 R 114. 1.1 0:07.29 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
26012 samzapo 20 0 619M 543M 18664 R 114. 0.8 0:06.14 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
26024 samzapo 20 0 587M 504M 18544 R 114. 0.8 0:04.75 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
24505 samzapo 20 0 2219M 2156M 22340 R 105. 3.4 0:40.21 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
24526 samzapo 20 0 2426M 2358M 23080 R 105. 3.7 0:40.18 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
24547 samzapo 20 0 2308M 2248M 22744 R 105. 3.5 0:40.21 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
24899 samzapo 20 0 1865M 1803M 18992 R 105. 2.8 0:30.36 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
24903 samzapo 20 0 1620M 1547M 22468 R 105. 2.4 0:29.81 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
25360 samzapo 20 0 1511M 1445M 18820 R 105. 2.3 0:21.06 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
25578 samzapo 20 0 1248M 1182M 18728 R 105. 1.8 0:17.50 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
25733 samzapo 20 0 919M 852M 18768 R 105. 1.3 0:12.58 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
25798 samzapo 20 0 963M 884M 19072 R 105. 1.4 0:10.04 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
25996 samzapo 20 0 557M 475M 18704 R 105. 0.7 0:07.51 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
26000 samzapo 20 0 681M 634M 18532 R 105. 1.0 0:07.35 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
26008 samzapo 20 0 611M 564M 18532 R 105. 0.9 0:06.73 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
26016 samzapo 20 0 582M 508M 18656 R 105. 0.8 0:06.08 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
26089 samzapo 20 0 409M 363M 18328 R 105. 0.6 0:03.92 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
26097 samzapo 20 0 436M 344M 18356 R 105. 0.5 0:03.17 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
26127 samzapo 20 0 331M 305M 17996 R 105. 0.5 0:03.00 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
26208 samzapo 20 0 308M 208M 9044 R 105. 0.3 0:01.51 /usr/lib/gcc/x86_64-linux-gnu/11/cc1plus -quiet -I /home/samzapo/Develop
26212 samzapo 20 0 129M 120M 4320 R 105. 0.2 0:00.56 htop
What is wrong with my scheduler and how can I fix it?
Some additional information:
$ uname -a
Linux <hostname> 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
my /etc/default/grub isolates a few CPUs in the above examples:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=0,1,2,3,4,5,6,7,8"
