Inconsistent disk usage/free space available depending of tools used

Question

Problem

After reading many posts/forum threads I still don't understand if my hard drive has some space left or not...

I started looking into it because I installed Syncthing today and a notification error says that my disk is almost full and syncthing can't run, but I thought that I had about 40 Go left...

What I did to try to understand it and solve it

• ran Disk analyzer (on root of system) : says 441,3 Go occupied / 45,4 Go available on total of 470,9 Go on a 480 GB disk
• ran Disk analyser as sudo (on root of system) : says 443,6 Go occupied / 45,4 Go available on total of 470,9 Go on a 480 GB disk

Note : I don't understand why only the "occupied "size is different... And I suppose that the 10 Go difference between 470 and 480 is due to reserved memory for the system or something like that (?).

• ran df

which outputs /dev/sda5 459849800 433180284 3236916 100% / so says its full...

• ran sudo du -h --max-depth=1 /

which outputs :

48G /var
0   /sys
4,0K    /srv
4,0K    /mnt
72M /root
63G /snap
4,0K    /cdrom
1,9G    /opt
22M /etc
du: impossible de lire le répertoire '/proc/3353/task/3353/net': Argument invalide
du: impossible de lire le répertoire '/proc/3353/net': Argument invalide
du: impossible de lire le répertoire '/proc/6203/task/6203/net': Argument invalide
du: impossible de lire le répertoire '/proc/6203/net': Argument invalide
du: impossible d'accéder à '/proc/80737/task/80737/fd/3': Aucun fichier ou dossier de ce nom
du: impossible d'accéder à '/proc/80737/task/80737/fdinfo/3': Aucun fichier ou dossier de ce nom
du: impossible d'accéder à '/proc/80737/fd/4': Aucun fichier ou dossier de ce nom
du: impossible d'accéder à '/proc/80737/fdinfo/4': Aucun fichier ou dossier de ce nom
0   /proc
325G    /home
16K /lost+found
0   /dev
368K    /tmp
272M    /boot
21G /usr
8,0K    /media
du: impossible d'accéder à '/run/user/1000/gvfs': Permission non accordée
du: impossible d'accéder à '/run/user/1000/doc': Permission non accordée
4,0M    /run
460G    /

so says there are about 10G free on the 470 Go total.

• installed ncdu which like du says that disk usage is 459,2 GiB

• ran lsof -nP +L1

which outputs 1413 lines of "deleted" files (vast majority are "memfd:mozilla-ipc"...) of various size. But I didn't find how to sum all those files to check the total disk usage.

• ran find /proc/[0-9]*/fd -lname '*(deleted)' 2>/dev/null | perl -lne '($l = readlink) =~ s/ (deleted)$//; print -s, " $_ $l"' | sort -g (command found on another askubuntu thread) to sort the result by file size

which outputs 1516 lines, and here are the top ten lines - I suppose first number is size in bytes (?) :

2482816 /proc/2758/fd/378 /memfd:gdk-wayland (deleted)
5439888 /proc/2758/fd/322 /memfd:gdk-wayland (deleted)
6031750 /proc/22782/fd/28 /tmp/.org.chromium.Chromium.7Uj5wU (deleted)
6987776 /proc/5122/fd/30 /memfd:mozilla-ipc (deleted)
8087040 /proc/2758/fd/351 /memfd:gdk-wayland (deleted)
8294400 /proc/2758/fd/224 /memfd:gdk-wayland (deleted)
8783424 /proc/2758/fd/327 /memfd:gdk-wayland (deleted)
8783424 /proc/2758/fd/360 /memfd:gdk-wayland (deleted)
9216000 /proc/2758/fd/331 /memfd:gdk-wayland (deleted)
67108864 /proc/2671/fd/6 /memfd:pulseaudio (deleted)

Questions

• I am in the state where I don't know if my disk is really full or not... there are a lot on inconsistencies in the numbers (I'm not saying the numbers are wrong, I suppose that they do not always mean the same thing, but this is non-intelligible to me). So if anyone can help me understand that point, I'll be grateful !
• I do not know if it's full because of "deleted" files that eat space or not. Any help on how to measure the disk usage of deleted files would also be appreciated
• FinalIy I don't understand why I see 63G /snap with du while the Disk analyzer says its only 188,4 ko. I understand that there are symlinks to snaps in this folder and it looks like those symlinks (to /var/lib/snapd/snaps/) account for 24 G says du. Are those 24 G also in the 63 G of /snap ? I checked the du output of /snap and it doesn't look like so... But adding 63 G to the Disk analyzer output results in a total above the real disk size... This totally lost me...

Please tell me if you need more information, I'll be glad to give it to you :)

Edits

Full df output (numbers for /dev/sda5 differs because I have made some room...) :

Sys. de fichiers blocs de 1K   Utilisé Disponible Uti% Monté sur
tmpfs                1625812      2204    1623608   1% /run
/dev/sda5          459849800 414179712   22237488  95% /
tmpfs                8129044     24768    8104276   1% /dev/shm
tmpfs                   5120         4       5116   1% /run/lock
/dev/sda1             523248         4     523244   1% /boot/efi
tmpfs                1625808      1720    1624088   1% /run/user/1000

Investigating further in order to understand better all this I realized that the Disks program on my system says that I have a sda2 partition of :

• 480 GB (479 564 137 472 bytes)

But those bytes are actually 480 gigabyte (10^9 bytes) but only 446.6 GB. The program also says sda5 partition has 47 GB free, which corresponds to the 46.8 Go given by the Disk Analyzer...

I never realized that disk storage space was given in "advantageous" gigabyte (10^9), and now better understand why I see a so "big" difference between the df output and Disk Analyzer.

Answer 1

You mixed up a lot of different things in your question, and that's probably why it's hard for you to understand what's going on.

Let's start from the last one, the "deleted" files in /proc. /proc is not a real directory on your disk; it's a virtual filesystem that reflects the state of processes in your system. The "files" in /proc/xxxx/fd represent file descriptors that are currently open by process number xxxx. File descriptors may refer to actual files on the disk, but may also refer to pretty much anything else, as in Unix-like systems almost everything that program can read data from or write data to is represented as a "file".

The command that you typed displays at the end of line the actual objects these file descriptors refer to. As you can see, most of them begin with /memfd: which means they are just memory blocks, used for communication between programs, that a program has allocated and then "deleted" as mentioned in the other answer (which effectively causes the memory block be automatically deallocated once the program stops using it). There is only one actual file there, which is /tmp/.org.chromium.Chromium.7Uj5wU.

If you're not a programmer interested in debugging any of the software that's running in your system, you basically don't have to care about the contents of /proc at all. Similarly for /sys, it's also a pseudo-directory used for communicating with the operating system's kernel, and /dev, which represents various devices present in your computer and recognized by the OS.

Which brings us to the next topic, output of your du command. It's misleading, as it includes "directories" that are just mount points for other filesystem than the one that interests you, ie. root filesystem placed on /dev/sda5 disk partition, as the output of your df command shows. (BTW. Does your df output only that one line referring to that partition that you posted? Normally, df should output a whole bunch of other lines too, referring to all filesystems.) One of the misleading entries in this output is /snap you are wondering about. All subdirectories under /snap are actually mount points for virtual filesystems contained within *.snap files in /var/lib/snapd/snaps directory and that directory is the place where they are actually taking disk space. Disk Analyzer seems to be counting only the size of actual subdirectory structure under /snap, ignoring these mounts, while du just counts them as if they were regular files in a regular diectory. So Disk Analyzer is correct on this one.

What you're missing is -x parameter to du - it tells du to ignore directories that belong to different file systems. This way you will get rid of misleading entries in du output.

BTW. If you want to know what is mounted where, just type the mount command - it will show you all filesystems and their mount points. You could see that there are multiple virtual filesystems (or "pseudo-filesystems") like sysfs, tmpfs, cgroup etc. mounted at various directories, you will also see the loop mounts of files in /var/lib/snapd/snaps to subdirectories of /snap directory.

Next, we come to your df output. It says that out of 459849800 Kbytes available on your root filesystem, 433180284 are used and 3236916 Kbytes are free. However, the sum 433180284+3236916 doesn't equal 459849800, it's only 436417200. Why is this? As mentioned in the other answer, about 5% of disk space is reserved for root processes, which means processes running as root can use the remaining 459849800-436417200=23432600 Kbytes, while processes running as regular users cannot. Out of the space available to user processes (436417200 Kbytes), 433180284 Kbytes are already used which is 99,2% - df apparently rounds this up to 100%.

In case of any doubt and inconsistent numbers, you should rely on what df shows you, as it displays the same data that operating system actually "sees" when it tries to allocate space for a new file. If df would show 0 Kbytes free, then you won't be able to write anything to the disk. In your case, you still have about 3 Gbytes free. So df is the ultimate answer to the question if your disk is full or not.

Finally, why Disk Analyzer running as root shows you different occupied size than running as regular user? Simply, it's a permissions thing. If running as non-root, the program simply cannot access some directories, so it cannot count the size of files in them. So these files are missing from the "occupied" count. In that case Disk Analyzer should show you a prominently visible warning at the top of a window that it couldn't scan some of the directories.

What I don't understand, however, is why it shows 45G free while this isn't the case actually. From my experiments it seems that Disk Analyzer is taking into account the free space reserved for root (which df omits), but in your case the sum of space available for root and available for user programs is about 26G, not 45. I also noticed that Disk Analyzer is apparently displaying the numbers using decimal gigabytes (1000x1000x1000 bytes), while tools like du and df when run with -h parametr use binary gigabytes (1024x1024x1024 bytes), so the numbers in Disk Analyzer will be slightly larger (by the factor of around 1,07), but this still won't change 26G to 45G. So I don't understand where did 45G come from. Maybe it's some bug in Disk Analyzer, maybe someone else could explain it.

Answer 2

"Disk space" is a per-filesystem concept, not per-disk or per-partition.

As a leftover from the days of tiny disks, in order to prevent sysadmins all-nighters, 5% of disk space is reserved for root (UID 0) processes. So,sudo df | grep -v loop and df | grep -v loop will get different results.

(deleted) files are files used by processes for temporary use (e.g. cache). The general method is:

1. Create the file on disk
2. Initialize the file, using up disk space.
3. Open the file, increasing the "use count" in the file's inode.
4. unlink() (delete) the file, which clears the directory entry for the file, but doesn't release the disk space, since the "use count" was incremented in step 3.
5. Now, the process has a private (accessible only by the process and it's children) file on disk.
6. When the file is close()ed (all open files are closed when the process exits), the "use-count" in the inode is decremented. If the "use-count" is zero, the disk space is released.

This is a programner's way of managing temporary files that automatically deletes them when the program ends. The other, bad way requires the programmer to keep track of all the temporary files and catch all possible program failures to do the cleanup.