10

We use upstart to manage our services on our Ubuntu servers. They produce logs which are logged out to /var/log/upstart/SERVICE_NAME.log

Then daily, the log files are rotated using the logrotation script which comes with 12.04 LTS:

/var/log/upstart/*.log {
        daily
        missingok
        rotate 7
        compress
        notifempty
        nocreate
}

The problem is that while logrotate moves the files, it doesn't appear to signal to upstart to close and reopen the files, leaving the upstart process writing to a delete PID.

init          1       root    8w      REG              202,1        64       2431 /var/log/upstart/dbus.log.1 (deleted)
init          1       root   13w      REG              202,1        95       2507 /var/log/upstart/acpid.log.1 (deleted)
init          1       root   14w      REG              202,1       127      17377 /var/log/upstart/whoopsie.log.1 (deleted)
init          1       root   36w      REG              202,1       122       6747 /var/log/upstart/SERVICE_NAME.log.1 (deleted)
init          1       root   37w      REG              202,1        30       6762

Obviously I could redirect the output from my own services to other log files, but the problem would still be there for the system processes. Also I would rather not have to build more infrastructure than what I need.

Ztyx
  • 153
Andrew Newdigate
  • 201
  • 1
  • 6

2 Answers2

2

I believe you have 3 options.

  1. You modify the existing configuration by adding "copytruncate"

    /var/log/upstart/*.log { copytruncate daily missingok rotate 7 compress notifempty nocreate }

  2. If you can't or (are not allowed) to change the existing logrotate config because of other log files that don't suffer and the existing config works for them, then move your "SERVICE_NAME.log" files to a new folder under /var/log if your wish, create a new config with the "copytruncate" and add it to the cron.daily.

  3. a) If you're not allowed to change the host os logrotate config or add to the host OS's cron.daily, then your third option is to change the scripts or programs to either check that the file exists before writing out to the file. b) Another way is a bit of point 2 above which is to move your logfiles somewhere else and within you script or program, execute the logrotate command specific for that program's logfile.

Point 3b above is more tricky but more elegant and it is what I use most of the time as it means the program is self-sufficient and self-managed and doesn't need the OS's jobs to babysit it.

To find out how to manually run logrotate and add it to your program or script just type:

man logrotate

or

logrotate --help

If you are using Python for your programs you can check out how this program uses it to self-manage its log files. http://bazaar.launchpad.net/~ferncasado/keep.awake/trunk/files/head:/v4/

DanglingPointer
  • 1,243
0

Turns out, this is a known problem and the ticket remains open as I type this.

The right thing to do is, probably, to just remove the /etc/logrotate.d/upstart altogether and rotate the files of the individual services individually. Because the directory (/var/log/upstart/) contains only the stdout/stderr of the various services -- and no service meant to run as a daemon should be outputting to those two channels at all. Except, maybe, at the very startup.

On the systems I'm managing, three services are run by upstart: php5.6-fpm, php7.1-fpm, and acpid. None of the three logs are active but sometimes the fpm is restarted due to its main log-file (/var/log/php5.6-fpm.log) being rotated -- and it causes this noise, because it outputs some inanity at startup.

If you insist on rotating these files anyway, you can rely on the fact, that their names match the names of the services and use the following postrotate script:

    postrotate
            service=${1##*/}
            service=${service%.log*}
            service $service restart > /dev/null
    endscript

For the above to work, be sure to not use the sharedscripts verb in there -- my scriptlet relies on the fact, the actual path to the file will be passed to it as the first argument ($1).

(The redirection into /dev/null is useful, because service-command is noisy -- and you don't want such noise e-mailed to you by cron. Notice, that I'm not redirecting stderr there, only stdout -- if there is a problem, you'll still get your e-mail about it.)

Mikhail T.
  • 521