29

The disk that contains Ubuntu on my computer is 115GB in size.

When the disk was 114.7GBfull.

I started deleting some files here and there to free up around 5GB.

But I noticed the disk was 114GB filled again. I thought it was some cache or swap thingy so didn't give it much thought and went ahead and freed around 40GBby shifting some media files out of the disk.

I have a notification in an hour or so that the disk is full yet again!! 40GB!!! All gone!

So I debugged the problem down to the /var/log/syslog file which was initially definitely less than 20GB. I know this because /var folder was 17GB before I cleared the memory.

/var/log/syslog file is currently 55.9GB!

Could someone be so gracious to explain this anomaly? Is this a bug? Or am I affected with some malware or virus?

ant_1618
  • 811
  • 2
  • 8
  • 11

8 Answers8

33

This indefinite growth generally occurs due to repeated log of one or more errors from the same source. In my case it was due to continual report of connection error from the wifi monitoring interface mon0 I have used to monitor my wifi traffic. There have been error reports of such overflow occurring in various other interfaces like tun0 from VPN etc.

I have resolved my issue by clearing the /var/log/syslog file

To tackle this error

  1. You need to find the source of this error and stop it from producing any further overflow of log
  2. Then clear the var/log/syslog file

Problems you might face doing the same

  1. Cant open var/log/syslog: due to massive size any editor is bound to crash
  2. Cant clear /var/log/syslog: Again due to massive size clearing is a challenge too

So, for viewing the error that caused the overflow

tail -f /var/log/syslog

For clearing use:

sudo sh -c 'cat /dev/null > /var/log/syslog'
Fábio Batista
  • 125
ant_1618
  • 811
  • 2
  • 8
  • 11
13

I had similar issues, my syslog file had 115GB and syslog.1 another 115G, plus multiple compressed files.

1st step find the source:

watch tail /var/log/syslog

you will probably notice the common erro entries; after that; assuming that your files are too big; it is ALMOST pointless to rotate. So, you can delete all the compressed files and files *.1 to recover disk space (my case about 300GB)

2nd step TRUNCATE THE FILE, DO NOT DELETE (or you may have a lot of problems with permissions in the future), there are many methods, including:

sudo tee /var/log/syslog </dev/null

you can even do the second step before and keep watching to find the cause, but be sure, if you don't it will happen again. Probably it is something in a loop, system services is a good place to start looking (something that restart very fast and always for exemple)

Thiago Conrado
  • 408
3

Check the /var/lib/logrotate/status and make sure that it's getting rotated properly. You also need to view the contents of the file and see if it's a system issue throwing alarms constantly.

Tom Damon
  • 477
3

Try this. It should work correctly and clean it up:

sudo sh -c 'cat /dev/null > /var/log/syslog'
anonymous2
  • 4,325
Ahmed Kamel
  • 39
  • 1
0

Step 1: Create a script file example clearlog.sh in location /var/log/ with file permission 'chmod 775 clearlog.sh'

clearlog.sh file with below lines:

#!/bin/sh
LINECOUNTS=(`cat /var/log/syslog | wc -l`)

if [ $LINECOUNTS -gt "1000" ];
    then
        echo "Recreating syslog file due to lines "$LINECOUNTS" are more than 1000 lines"
        truncate -s 0 /var/log/syslog
        sleep 5
else
        echo "syslog file having line count "$LINECOUNTS" less than 1000}"
fi
/bin/systemctl restart syslog
exit

Step 2: Edit the file in location /etc/logrotate.d/logs add postrotate section

/var/log/syslog {
        daily
        rotate 3
        missingok
        copy
        notifempty
        compress
        dateext
        dateformat -%Y%m%d%H%M
        postrotate
                /var/log/clearlog.sh
        endscript
}

To test above script from command terminal execute this cli 'logrotate -f /etc/logrotate/logs'

Manikandan Jayaraman
  • 1
0

A late response to the original post but this helped me. When running a program as a service it seems that all printfs are diverted to syslog rather than stdout, if you have a lot of printfs then this will fill up the syslog very quickly. I had some issues with some of the methods suggested here but

sudo tee /var/log/syslog </dev/null

worked every time.

Note that (for me) I when the syslog file filled up enough it prevented the pi desktop from starting up - I worked around this by logging in through Putty and using the tee command first then rebooting.

Steve M
  • 1
0

If you don't want to trackdown the offending app that writes to syslog too often/much/repeatedly/..., then modify your logrotate configuration. You should have /etc/logrotate.d/rsyslog file (or some similar config file. Add one of the following in the config (right below "rotate" config... or elsewhere if you know what you are doing):

size 100k <= set this to rotate logs when it reaches specified bytes large minsize 100k <= set this to prevent time-based log rotate unless log size become this large maxsize 100k <= set this to rotate log that gets this big in addition to any time-based rotation.

The 100k size is just for example - depends on how much log you want to see at any given time... I personally like minsize setting the best, but that's just me. And the 100k is just for example. Set it to a sane value for your disk space available (and your debugging habit/preference).

Marcus Yoo
  • 101
-1

Installing shurfshark snap made my syslog max out my hard drive in a day. Uninstalled and it appears to be stable in size now.

mkd
  • 9