7

my current system: 14.04.4 LTS (GNU/Linux 3.13.0-85-generic x86_64)

$ free
             total       used       free     shared    buffers     cached 
Mem:      16366288   16090588     275700    4047472    1019652  5253144
-/+ buffers/cache:    9817792    6548496 Swap:      7813116    7308592     504524



$ cat /proc/sys/fs/file-nr
1196103  0   1624594

For a couple of months (and a number of updates) I've got the problem that suddenly after some days of working with my box (and daily suspending it) memory usage and the number of allocated file descriptors shoots up.

All of a sudden the "committed" memory rises, up to around 120 - 160 GB(!). Normally this value sits around my physical RAM. Another symptom is that all my RAM + swap is used up. The result is an unusable system. For now I only could fix this situation by rebooting. Killing nearly all processes after having shut down X did not help to free up the committed value.

A strong indicator seems to be the unreasonably high value of the open files counter in /proc/sys/fs/file-nr -- this goes up above 2 million. Lately I tried to limit this to 1624594 but the only result is (of course) that I don't run out of memory but instead out of FDs.

Having Munin in place I can see a strong temporal correlation between FDs and memory usage.

I have some CIFS shares mounted and my user has its gvfsd-fuse. Root FS is ext4.

The results of lsof and traversing /proc/ list only about 10% of /proc/sys/fs/file-nr.

I suspected MTP to be the cause (when I plug in my Motorola phone) but recently this happened without having used MTP since the last reboot.

Sidenote: I've got radeon drivers for video:

ii  libdrm-radeon1:amd64                                        2.4.64-1~ubuntu14.04.1                        
ii  libdrm-radeon1:i386                                         2.4.64-1~ubuntu14.04.1                        
ii  radeontool                                                  1.6.3-1                                       
ii  xserver-xorg-video-radeon                                   1:7.3.0-1ubuntu3.1

UPDATE

root:/proc$ (echo -n "0" ; for processid in [0-9]*; do echo -n "+$(ls /proc/$processid/fd/ | wc -l)"; done;echo)|bc
2597
root:/proc$ lsof -n|wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
84504
root:/proc$ sudo -u mdo lsof -n|wc -l
72170

How can I find out what's eating all these FDs and my memory?

mdo
  • 243

1 Answers1

5

This will show the number of open files per process ID:

cd /proc
for processid in [0-9]*
do
    echo "Process ID = $processid: $(ls /proc/$processid/fd/ | wc -l) file descriptors"
done

(save and execute with sudo).

Then you can track down what that process ID belongs to.

Rinzwind
  • 309,379