95

When running tail -f filename, I got the following message:

tail: inotify cannot be used, reverting to polling: Too many open files

Is that a potential problem?

How do I diagnose what's responsible for all the open files? I have a list of suspect processes, but if they don't turn out to be the culprits, instructions that don't rely on knowing which process to check would be useful.

Andrew Grimm
  • 1,354

3 Answers3

87

You can use lsof to understand who's opening so many files. Usually it's a (web)server that opens so many files, but lsof will surely help you identify the cause.

Once you understand who's the bad guy you can

  • kill the process/stop the program
  • raise the ulimit

If output from lsof is quite huge try redirecting it to a file and then open the file

Example (you might have to Ctrl+C the first command)

lsof > ~/Desktop/lsof.log
cat ~/Desktop/lsof.log | awk '{ print $2 " " $1; }' | sort -rn | uniq -c | sort -rn | head -20
vim ~/Desktop/lsof.log
gatorback
  • 6,523
Andrea Olivato
  • 1,313
35

In case anyone else needs it...

ulimit -a

Will display all current limits. Specifically ulimit -n 70000 will set the file descriptor limit.

Also...

cat /proc/sys/fs/file-max

Will display/set the kernel limit if edited.

sudo echo 200000 > /proc/sys/fs/file-max

A much more detailed explanation can be found at...

How do I increase the open files limit for a non-root user?

Community
  • 1
kervin
  • 1,451
10

While ulimit can be used to determine how many files are allowed to be open per process you may want to find the culprit.

@itsadok @Tyler Collier @gaoithe in comments to other answers highlight that sorting and counting which process has the most files open is the best thing to do here:

sudo lsof | head -1 | awk '{ print "COUNT " $2 " " $1; }' && sudo lsof +c 0 | awk '{ print $2 " " $1; }' | sort -rn | uniq -c | sort -rn | head -20

Above command:

  • Gives output a header
  • lsof gives list of open files
  • +c 0 option instructs lsof to print full command
  • awk '{ print $2 " " $1; }' prints PID and COMMAND column in results
  • sort -rn sorts results so identical entries are next to each other (needed for uniq -c to work properly)
  • uniq -c counts files open by PID/command
  • sort -rn sorts results by count
  • head -20 shows top 20 files open by PID/command

Note: This will count includes "files" that don't count towards limits

You may want to investigate even further by looking at limits for a PID, # of files open for a specific PID, and limit lsof to only count files that count towards limit - see https://serverfault.com/a/964752/152562.

bbodenmiller
  • 239