Warning
This answer is specific to Ubuntu version 20.04 LTS (and later). This approach will only work on LTS versions of Ubuntu that either came with the HWE kernels installed, or if the HWE kernels were installed at a later date.
Note: It is advised to keep Ubuntu Desktop 20.04 LTS with the kernel flavor picked during installation. This is for HWE flavor. Changing to track GA kernel may result in regressions of performance, hardware support, and certified features.
Note 2: This answer is not for OEM kernels.
Make sure you have the 5.4 kernels and headers
Open a terminal and use this command:
sudo apt install --install-recommends linux-generic
If everything is already installed, then you will see a message saying so. Otherwise some packages will get installed.
If NVIDIA drivers are required check appropriate level and install them:
ubuntu-drivers list
And then install appropriate NNN version of the drivers:
sudo apt install --install-recommends linux-modules-nvidia-NNN-generic
Once you have the latest kernel version 5.4 installed, use the Advanced menu in GRUB to boot from the latest 5.4 kernel and make sure everything works as expected.
Purge the HWE kernels
While you are booted to the latest 5.4 kernel you can remove the 5.13 kernels and headers using the following command:
sudo apt remove --purge linux-generic-hwe-20.04
This will stop the system getting kernel 5.13 updates. This does not remove the the kernels, headers, modules, and modules-extra packages HWE versions.
The following command will remove all the 5.13 kernel , headers, modules, and modules-extra packages:
sudo apt remove --purge linux-hwe-* linux-modules-5.1* linux-modules-5.8.0-* linux-modules-5.6.0-*
Be careful with the * in this command.
Reference: Ubuntu Kernel LTS Enablement Stack Wiki
Hope this helps