I've discovered, that even after doing all the cleanup listed with OP, there is still this - 192.168.2.31 having been assigned to eth0 at some point in the past (and which I do not want anymore):
$ sudo grep -r '192.168.2.31' /var/ 2>/dev/null
/var/lib/NetworkManager/internal-57fb77c9-12a8-32c7-b276-90a8d1904533-eth0.lease:ADDRESS=192.168.2.31
So in this case, I'd have to find how to force NetworkManager to forget all its leases, but cannot find anything yet ...
EDIT: https://bugs.launchpad.net/ubuntu/+source/netplan.io/+bug/1979674
Although sudo systemctl status NetworkManager states that NetworkManager's own DHCP client is being used
Oh, so NetworkManager has its own DHCP client, it does not use dhclient? Oh, what a crock of shit all of this is ...
EDIT: How to confirm this from logs? There is no more /var/log/syslog ... Where are the NetworkManager logs? :
The answers are now obsolete (2017), the logs are now here ... If you want to limit the amount of logs to the time the system has been booted instead, use the -b option: journalctl -b 0 -u NetworkManager
Ok, so I have:
$ journalctl -b 0 -u NetworkManager | grep dhcp | grep 'init\|eth0' | head -5
May 02 15:18:23 raspberrypi NetworkManager[717]: <info> [1714655903.2163] dhcp: init: Using DHCP client 'internal'
May 02 15:18:27 raspberrypi NetworkManager[717]: <info> [1714655907.3436] dhcp4 (eth0): activation: beginning transaction (timeout in 45 seconds)
May 02 15:22:27 raspberrypi NetworkManager[717]: <info> [1714656147.7046] dhcp4 (eth0): canceled DHCP transaction
May 02 15:22:27 raspberrypi NetworkManager[717]: <info> [1714656147.7048] dhcp4 (eth0): activation: beginning transaction (timeout in 45 seconds)
May 02 15:22:27 raspberrypi NetworkManager[717]: <info> [1714656147.7048] dhcp4 (eth0): state changed no lease
EDIT: Get a list of network manager network connections with Bash?
To get a list of connections: nmcli connection
Ok, so for me:
$ nmcli connection
NAME UUID TYPE DEVICE
SomeNet d4e4973b-45b7-4a11-8a7d-b408c1125b26 wifi wlan0
lo 2166fa1e-34fd-43e1-a0e5-22b4d007ea2c loopback lo
OtherNet 9fc89e3d-3566-4851-ae79-994a071665d3 wifi --
Wired connection 1 57fb77c9-12a8-32c7-b276-90a8d1904533 ethernet --
At least, it is the same UUID as in that /var/lib/NetworkManager/internal-*eth0.lease:ADDRESS=* file above; there are also ipv4.dhcp* settings there, but not much for leases:
$ nmcli connection show "Wired connection 1" | grep ipv4.dhcp
ipv4.dhcp-client-id: --
...
EDIT: I found https://unix.stackexchange.com/questions/284550/check-my-dhcp-lease - I get absolutely nothing from the suggestions there:
$ nmcli -f ipv4.method connection show "Wired connection 1"
ipv4.method: auto
$ nmcli -f DHCP4 connection show "Wired connection 1"
$
$ cat /run/NetworkManager/devices/1
[device]
managed=true
perm-hw-addr-fake=00:00:00:00:00:00
connection-uuid=2166fa1e-34fd-43e1-a0e5-22b4d007ea2c
nm-owned=false
$ ls -la /run/systemd/netif/leases/
total 0
drwxr-xr-x 2 systemd-network systemd-network 40 May 2 15:17 .
drwxr-xr-x 5 systemd-network systemd-network 100 May 2 15:17 ..
EDIT: some more explanation in Re: Where does NetworkManager keep DHCP client lease data?:
the files in /var/lib/NetworkManager are internal not public API. ... These files are not supposed to give information about the lease.
You can see the current lease information on the D-Bus API, for example via [...] nmcli -f DHCP4 device show ens1. In recent NM versions, you can also find the same information in /run/NetworkManager/devices/$IFINDEX (if you prefer reading files). This is public API.
A lease only makes sense to NetworkManager while it's running and connected. It just knows in memory how long the lease is valid. ... If you re-activate the device (or reboot or restart NM), it will try to get a new lease and start the DHCP state machine again. It only uses the previous ADDRESS= to give a hint to the DHCP server which address it would like to have.
If the device is currently not active, then there is no public API for past leases. Yes, you could parse the
/var/lib/NetworkManager/*.lease to get some some of the information, but this only contains information which NetworkManger cares about (the ADDRESS=), not most of the interesting data.
Oh, dear ... so what does this mean, that to forget the lease I should just delete /var/lib/NetworkManager/internal*lease*, and then restart NetworkManager, is that it?
EDIT: I've found references to nmcli device reapply in https://www.cyberciti.biz/faq/howto-linux-renew-dhcp-client-ip-address/ and https://www.kc8apf.net/2019/03/renewing-dhcp-while-keeping-a-networkmanager-connection-up/ - apparently, I'd not be able to use that in my eth0, as it is unconnected at the moment:
$ nmcli device
DEVICE TYPE STATE CONNECTION
wlan0 wifi connected SomeNet
lo loopback connected (externally) lo
eth0 ethernet disconnected --
p2p-dev-wlan0 wifi-p2p disconnected --
$ sudo nmcli con down id "Wired connection 1"
Connection 'Wired connection 1' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/56)
$ sudo nmcli con up id "Wired connection 1"
Error: Connection activation failed: IP configuration could not be reserved (no available address, timeout, etc.)
Hint: use 'journalctl -xe NM_CONNECTION=57fb77c9-12a8-32c7-b276-90a8d1904533 + NM_DEVICE=eth0' to get more details.
$ sudo nmcli device reapply eth0
Connection successfully reapplied to device 'eth0'.
$ sudo bash -c 'ls -la /var/lib/NetworkManager/internaleth0'
-rw-r--r-- 1 root root 59 Apr 25 04:11 /var/lib/NetworkManager/internal-57fb77c9-12a8-32c7-b276-90a8d1904533-eth0.lease
Yup, doesn't look like it "forgot" the lease like this, either ...
EDIT: Just to note: in the end, I did delete the /var/lib/NetworkManager/internal-*-eth0.lease (see comment below by @ulidtko on the right way to do that); the interesting thing is that, after that I ran dhclient, restarted NetworkManager, and so on - and I could even connect - however, an interesting thing is that I did not get another /var/lib/NetworkManager/internal-*-eth0.lease file ?! SO how come it was there in the first place? Well, while I remember, let me note: that 192.168.2.31 was an address I was testing in tftpd64 DHCP server running on Windows, and IIRC, there was a checkbox in tftpd64 "Persistent Lease" which was enabled - and as I understand it, "persistent lease" is a lease that never expires; and apparently NetworkManager saves those persistent lease addresses in these internal* files? If so, that (the "persistent lease") also explains why that file did not just disappear after reboot ...