9

I have a (formerly working) logrotate setup for OpenResty that I took from a previous Ubuntu 18.04 installation. Yet the logrotate.service fails with this error now ...

error: error renaming /usr/local/openresty/nginx/logs/access.log.60.zst
to /usr/local/openresty/nginx/logs/access.log.61.zst: Read-only file system

... and I have difficulties understanding why. The new machines are running Ubuntu 20.04, but I don't see why that should make a difference in this case.

First, here is the config:

$ cat /etc/logrotate.d/custom-openresty

/usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
{
  daily
  rotate 60
  maxsize 1G
  missingok
  notifempty
  compress
  compresscmd /usr/bin/zstd
  uncompresscmd /usr/bin/unzstd
  compressoptions -9 --long -T1
  compressext .zst
  delaycompress
  sharedscripts
  postrotate
    test ! -f /usr/local/openresty/nginx/logs/nginx.pid || kill -USR1 cat /usr/local/openresty/nginx/logs/nginx.pid
 endscript
}

/etc/logrotate.conf is unchanged and looks like this:


# see "man logrotate" for details
# rotate log files weekly
weekly

use the adm group by default, since this is the owning group


of /var/log/syslog.


su root adm


keep 4 weeks worth of backlogs


rotate 4


create new (empty) log files after rotating old ones


create


use date as a suffix of the rotated file


#dateext


uncomment this if you want your log files compressed


#compress


packages drop log rotation information into this directory


include /etc/logrotate.d


system-specific logs may be also be configured here.

Here is the state of my files (I would expect to get access.log.1 after logrotate ran):

$ ls -alhg /usr/local/openresty/nginx/logs/
total 10G
drwxr-xr-x  2 root 4.0K Sep 16 20:04 .
drwxr-xr-x 18 root 4.0K Sep 16 19:42 ..
-rw-r--r--  1 root  10G Sep 16 19:56 access.log
-rw-r--r--  1 root 5.5K Sep 16 19:56 error.log

However, the logrotate.service fails with this error:

~$ systemctl status logrotate.service 
● logrotate.service - Rotate log files
     Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2020-09-16 20:11:32 UTC; 4min 32s ago
TriggeredBy: ● logrotate.timer
       Docs: man:logrotate(8)
             man:logrotate.conf(5)
    Process: 27403 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE)
   Main PID: 27403 (code=exited, status=1/FAILURE)

$ sudo journalctl --unit logrotate.service


Sep 16 20:11:32 fetcher-scheduler systemd[1]: Starting Rotate log files...
Sep 16 20:11:32 fetcher-scheduler logrotate[27403]: error: error renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst: Read-only file system
Sep 16 20:11:32 fetcher-scheduler systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Sep 16 20:11:32 fetcher-scheduler systemd[1]: logrotate.service: Failed with result 'exit-code'.
Sep 16 20:11:32 fetcher-scheduler systemd[1]: Failed to start Rotate log files.

When I run it (without systemd) in debug mode as root, I get the following output:

# logrotate -v -d /etc/logrotate.d/custom-openresty

reading config file /etc/logrotate.d/custom-openresty
compress_prog is now /usr/bin/zstd
uncompress_prog is now /usr/bin/unzstd
compress_options is now  -9 --long -T1
compress_ext is now .zst
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries
Creating new state
Creating new state
...
Creating new state


Handling 1 logs


rotating pattern: /usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
 after 1 days (60 rotations)
empty log files are not rotated, log files >= 1073741824 are rotated earlier, old logs are removed
considering log /usr/local/openresty/nginx/logs/access.log
  Now: 2020-09-16 20:24
  Last rotated at 2020-09-16 20:11
  log needs rotating
considering log /usr/local/openresty/nginx/logs/error.log
  Now: 2020-09-16 20:24
  Last rotated at 2020-09-16 19:00
  log does not need rotating (log has been rotated at 2020-9-16 19:0, that is not day ago yet)
rotating log /usr/local/openresty/nginx/logs/access.log, log->rotateCount is 60
dateext suffix '-20200916'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
previous log /usr/local/openresty/nginx/logs/access.log.1 does not exist
renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst (rotatecount 60, logstart 1, i 60), 
renaming /usr/local/openresty/nginx/logs/access.log.59.zst to /usr/local/openresty/nginx/logs/access.log.60.zst (rotatecount 60, logstart 1, i 59), 
...
/logs/access.log.3.zst (rotatecount 60, logstart 1, i 2), 
renaming /usr/local/openresty/nginx/logs/access.log.1.zst to /usr/local/openresty/nginx/logs/access.log.2.zst (rotatecount 60, logstart 1, i 1), 
renaming /usr/local/openresty/nginx/logs/access.log.0.zst to /usr/local/openresty/nginx/logs/access.log.1.zst (rotatecount 60, logstart 1, i 0), 
log /usr/local/openresty/nginx/logs/access.log.61.zst doesn't exist -- won't try to dispose of it
renaming /usr/local/openresty/nginx/logs/access.log to /usr/local/openresty/nginx/logs/access.log.1
running postrotate script
running script with arg /usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
: "
    test ! -f /usr/local/openresty/nginx/logs/nginx.pid || kill -USR1 cat /usr/local/openresty/nginx/logs/nginx.pid
"

To me, everything looks normal. If I run it, it also works:

# logrotate -v /etc/logrotate.d/custom-openresty
reading config file /etc/logrotate.d/custom-openresty
compress_prog is now /usr/bin/zstd
uncompress_prog is now /usr/bin/unzstd
compress_options is now  -9 --long -T1
compress_ext is now .zst
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries
Creating new state
...
Creating new state

Handling 1 logs


rotating pattern: /usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
 after 1 days (60 rotations)
empty log files are not rotated, log files >= 1073741824 are rotated earlier, old logs are removed
considering log /usr/local/openresty/nginx/logs/access.log
  Now: 2020-09-16 20:26
  Last rotated at 2020-09-16 20:11
  log needs rotating
considering log /usr/local/openresty/nginx/logs/error.log
  Now: 2020-09-16 20:26
  Last rotated at 2020-09-16 19:00
  log does not need rotating (log has been rotated at 2020-9-16 19:0, that is not day ago yet)
rotating log /usr/local/openresty/nginx/logs/access.log, log->rotateCount is 60
dateext suffix '-20200916'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
previous log /usr/local/openresty/nginx/logs/access.log.1 does not exist
renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst (rotatecount 60, logstart 1, i 60), 
old log /usr/local/openresty/nginx/logs/access.log.60.zst does not exist
...
old log /usr/local/openresty/nginx/logs/access.log.2.zst does not exist
renaming /usr/local/openresty/nginx/logs/access.log.1.zst to /usr/local/openresty/nginx/logs/access.log.2.zst (rotatecount 60, logstart 1, i 1), 
old log /usr/local/openresty/nginx/logs/access.log.1.zst does not exist
renaming /usr/local/openresty/nginx/logs/access.log.0.zst to /usr/local/openresty/nginx/logs/access.log.1.zst (rotatecount 60, logstart 1, i 0), 
old log /usr/local/openresty/nginx/logs/access.log.0.zst does not exist
log /usr/local/openresty/nginx/logs/access.log.61.zst doesn't exist -- won't try to dispose of it
renaming /usr/local/openresty/nginx/logs/access.log to /usr/local/openresty/nginx/logs/access.log.1
running postrotate script

No error and in the end access.log.1 is created as expected:

# ls -algh /usr/local/openresty/nginx/logs/
total 10G
drwxr-xr-x  2 root 4.0K Sep 16 20:26 .
drwxr-xr-x 18 root 4.0K Sep 16 19:42 ..
-rw-r--r--  1 root  10G Sep 16 19:56 access.log.1
-rw-r--r--  1 root 5.5K Sep 16 19:56 error.log

Note that it said ...

log /usr/local/openresty/nginx/logs/access.log.61.zst
doesn't exist -- won't try to dispose of it

... instead of error renaming ... Read-only file system (as logrotate.service does)

Why does it work when running manually as root, but fails when it is executed by the logrotate.service?

I did not make any changes to logrotate.service. For completeness, here is the unit file:

$ systemctl cat logrotate.service
# /lib/systemd/system/logrotate.service
[Unit]
Description=Rotate log files
Documentation=man:logrotate(8) man:logrotate.conf(5)
ConditionACPower=true

[Service]
Type=oneshot
ExecStart=/usr/sbin/logrotate /etc/logrotate.conf


performance options


Nice=19
IOSchedulingClass=best-effort
IOSchedulingPriority=7


hardening options


details: https://www.freedesktop.org/software/systemd/man/systemd.exec.html


no ProtectHome for userdir logs


no PrivateNetwork for mail deliviery


no ProtectKernelTunables for working SELinux with systemd older than 235


no MemoryDenyWriteExecute for gzip on i686


PrivateDevices=true
PrivateTmp=true
ProtectControlGroups=true
ProtectKernelModules=true
ProtectSystem=full
RestrictRealtime=true

Now I'm running out of options. Any help in troubleshooting the problem is much appreciated.

Philipp Claßen
  • 4,884

2 Answers2

12

I think, I found it. The logrotate file is not the problem.

Instead, it is caused by the hardening features in the systemd unit file. After I disabled the ProtectSystem=full hardening options, it worked. The reason is that logrotate has to operate on the /usr directory in my case, which is readonly if the option is enabled.

From the documentation on ProtectSystem=

Takes a boolean argument or the special values "full" or "strict". If true, mounts the /usr and the boot loader directories (/boot and /efi) read-only for processes invoked by this unit. If set to "full", the /etc directory is mounted read-only, too. If set to "strict" the entire file system hierarchy is mounted read-only, except for the API file system subtrees /dev, /proc and /sys (protect these directories using PrivateDevices=, ProtectKernelTunables=, ProtectControlGroups=). This setting ensures that any modification of the vendor-supplied operating system (and optionally its configuration, and local mounts) is prohibited for the service. It is recommended to enable this setting for all long-running services, unless they are involved with system updates or need to modify the operating system in other ways. If this option is used, ReadWritePaths= may be used to exclude specific directories from being made read-only. This setting is implied if DynamicUser= is set. This setting cannot ensure protection in all cases. In general it has the same limitations as ReadOnlyPaths=, see below. Defaults to off.

To fix it properly, I added now the following line to /lib/systemd/system/logrotate.service:

ReadWritePaths=/usr/local/openresty/nginx/logs

Then ProtectSystem=full will mount everything as readonly except the directory where the logs are. Now, the readonly error is gone:

# systemctl daemon-reload && systemctl start logrotate
(completed successfully without output)

I assume, I never hit the problem on Ubuntu 18.04 because the hardening options were not yet the defaults. Ubuntu 20.04 has it now, but I don't know in what Ubuntu release they were first introduced.

Philipp Claßen
  • 4,884
2

Nice find to the same problem i was seeing. Editing nginx.conf to log to different directory (eg. the common /var/log/nginx) is an alternative that also works.

user1137190
  • 21