I'm part of a lab that has a shared scientific computing enviornment. My system administrator just told everyone in the lab that one of the nodes I am using is about to crash because /tmp is full. I have been trying to understand how I am using /tmp and am confused. I noticed that running lsof +D /tmp revealed a lot of open files that, according to the entries in the NAME column of the output, did not appear to be in /tmp or any of its subdirectories (and were not listed in the output of ls /tmp) and yet were being included in the command output. For example, here is part of the output:
(base) lsof -u cigoe +d /tmp 2>/dev/null
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
bash 81648 cigoe cwd DIR 253,2 4096 34238020 /home/scratch/cigoe/2022/proposal
bash 81648 cigoe rtd DIR 253,0 4096 128 / bash 81648 cigoe txt REG 253,0 1150648 3646 /usr/bin/bash
bash 81648 cigoe mem REG 253,0 2586930 3602 /usr/lib/locale/en_US.utf8/LC_COLLATE
bash 81648 cigoe mem REG 253,0 9253600 201909537 /var/lib/sss/mc/passwd
bash 81648 cigoe mem REG 253,0 46320 67889113 /usr/lib64/libnss_sss.so.2
...
I wasn’t able to get a good understanding of exactly how these open files were being associated with /tmp, but they seemed to collectively take up a lot of space (in terms of the sum of entries in the SIZE/OFF column, far too much to realistically fit on the device, so I’m not sure what’s going on there). In case it's relevant, our lab uses ZFS. I would appreciate help in answering the following question:
Most of the entries in the NAME column of the output from lsof +D /tmp do not begin with /tmp. Why is this the case?