logrotate succeeds when manually run as root, but fails with “Read-only file system” when run by logrotate.service

Spread the love
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  

I have a (formerly working) logrotate setup for OpenResty that I took from a previous Ubuntu 18.04 installation. Yet the logrotate.service fails with this error now …
error: error renaming /usr/local/openresty/nginx/logs/access.log.60.zst
to /usr/local/openresty/nginx/logs/access.log.61.zst: Read-only file system

… and I have difficulties understanding why. The new machines are running Ubuntu 20.04, but I don’t see why that should make a difference in this case.
First, here is the config:
$ cat /etc/logrotate.d/custom-openresty

/usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
{
daily
rotate 60
maxsize 1G
missingok
notifempty
compress
compresscmd /usr/bin/zstd
uncompresscmd /usr/bin/unzstd
compressoptions -9 –long -T1
compressext .zst
delaycompress
sharedscripts
postrotate
test ! -f /usr/local/openresty/nginx/logs/nginx.pid || kill -USR1 `cat /usr/local/openresty/nginx/logs/nginx.pid`
endscript
}

/etc/logrotate.conf is unchanged and looks like this:

# see “man logrotate” for details
# rotate log files weekly
weekly

# use the adm group by default, since this is the owning group
# of /var/log/syslog.
su root adm

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
#dateext

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# system-specific logs may be also be configured here.

Here is the state of my files (I would expect to get access.log.1 after logrotate ran):
$ ls -alhg /usr/local/openresty/nginx/logs/
total 10G
drwxr-xr-x 2 root 4.0K Sep 16 20:04 .
drwxr-xr-x 18 root 4.0K Sep 16 19:42 ..
-rw-r–r– 1 root 10G Sep 16 19:56 access.log
-rw-r–r– 1 root 5.5K Sep 16 19:56 error.log

However, the logrotate.service fails with this error:
~$ systemctl status logrotate.service
● logrotate.service – Rotate log files
Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2020-09-16 20:11:32 UTC; 4min 32s ago
TriggeredBy: ● logrotate.timer
Docs: man:logrotate(8)
man:logrotate.conf(5)
Process: 27403 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE)
Main PID: 27403 (code=exited, status=1/FAILURE)

$ sudo journalctl –unit logrotate.service

Sep 16 20:11:32 fetcher-scheduler systemd[1]: Starting Rotate log files…
Sep 16 20:11:32 fetcher-scheduler logrotate[27403]: error: error renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst: Read-only file system
Sep 16 20:11:32 fetcher-scheduler systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Sep 16 20:11:32 fetcher-scheduler systemd[1]: logrotate.service: Failed with result ‘exit-code’.
Sep 16 20:11:32 fetcher-scheduler systemd[1]: Failed to start Rotate log files.

When I run it (without systemd) in debug mode as root, I get the following output:
# logrotate -v -d /etc/logrotate.d/custom-openresty

reading config file /etc/logrotate.d/custom-openresty
compress_prog is now /usr/bin/zstd
uncompress_prog is now /usr/bin/unzstd
compress_options is now -9 –long -T1
compress_ext is now .zst
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries
Creating new state
Creating new state

Creating new state

Handling 1 logs

rotating pattern: /usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
after 1 days (60 rotations)
empty log files are not rotated, log files >= 1073741824 are rotated earlier, old logs are removed
considering log /usr/local/openresty/nginx/logs/access.log
Now: 2020-09-16 20:24
Last rotated at 2020-09-16 20:11
log needs rotating
considering log /usr/local/openresty/nginx/logs/error.log
Now: 2020-09-16 20:24
Last rotated at 2020-09-16 19:00
log does not need rotating (log has been rotated at 2020-9-16 19:0, that is not day ago yet)
rotating log /usr/local/openresty/nginx/logs/access.log, log->rotateCount is 60
dateext suffix ‘-20200916’
glob pattern ‘-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]’
previous log /usr/local/openresty/nginx/logs/access.log.1 does not exist
renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst (rotatecount 60, logstart 1, i 60),
renaming /usr/local/openresty/nginx/logs/access.log.59.zst to /usr/local/openresty/nginx/logs/access.log.60.zst (rotatecount 60, logstart 1, i 59),

/logs/access.log.3.zst (rotatecount 60, logstart 1, i 2),
renaming /usr/local/openresty/nginx/logs/access.log.1.zst to /usr/local/openresty/nginx/logs/access.log.2.zst (rotatecount 60, logstart 1, i 1),
renaming /usr/local/openresty/nginx/logs/access.log.0.zst to /usr/local/openresty/nginx/logs/access.log.1.zst (rotatecount 60, logstart 1, i 0),
log /usr/local/openresty/nginx/logs/access.log.61.zst doesn’t exist — won’t try to dispose of it
renaming /usr/local/openresty/nginx/logs/access.log to /usr/local/openresty/nginx/logs/access.log.1
running postrotate script
running script with arg /usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
: ”
test ! -f /usr/local/openresty/nginx/logs/nginx.pid || kill -USR1 `cat /usr/local/openresty/nginx/logs/nginx.pid`

To me, everything looks normal. If I run it, it also works:
# logrotate -v /etc/logrotate.d/custom-openresty
reading config file /etc/logrotate.d/custom-openresty
compress_prog is now /usr/bin/zstd
uncompress_prog is now /usr/bin/unzstd
compress_options is now -9 –long -T1
compress_ext is now .zst
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries
Creating new state

Creating new state

Handling 1 logs

rotating pattern: /usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
after 1 days (60 rotations)
empty log files are not rotated, log files >= 1073741824 are rotated earlier, old logs are removed
considering log /usr/local/openresty/nginx/logs/access.log
Now: 2020-09-16 20:26
Last rotated at 2020-09-16 20:11
log needs rotating
considering log /usr/local/openresty/nginx/logs/error.log
Now: 2020-09-16 20:26
Last rotated at 2020-09-16 19:00
log does not need rotating (log has been rotated at 2020-9-16 19:0, that is not day ago yet)
rotating log /usr/local/openresty/nginx/logs/access.log, log->rotateCount is 60
dateext suffix ‘-20200916’
glob pattern ‘-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]’
previous log /usr/local/openresty/nginx/logs/access.log.1 does not exist
renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst (rotatecount 60, logstart 1, i 60),
old log /usr/local/openresty/nginx/logs/access.log.60.zst does not exist

old log /usr/local/openresty/nginx/logs/access.log.2.zst does not exist
renaming /usr/local/openresty/nginx/logs/access.log.1.zst to /usr/local/openresty/nginx/logs/access.log.2.zst (rotatecount 60, logstart 1, i 1),
old log /usr/local/openresty/nginx/logs/access.log.1.zst does not exist
renaming /usr/local/openresty/nginx/logs/access.log.0.zst to /usr/local/openresty/nginx/logs/access.log.1.zst (rotatecount 60, logstart 1, i 0),
old log /usr/local/openresty/nginx/logs/access.log.0.zst does not exist
log /usr/local/openresty/nginx/logs/access.log.61.zst doesn’t exist — won’t try to dispose of it
renaming /usr/local/openresty/nginx/logs/access.log to /usr/local/openresty/nginx/logs/access.log.1
running postrotate script

No error and in the end access.log.1 is created as expected:
# ls -algh /usr/local/openresty/nginx/logs/
total 10G
drwxr-xr-x 2 root 4.0K Sep 16 20:26 .
drwxr-xr-x 18 root 4.0K Sep 16 19:42 ..
-rw-r–r– 1 root 10G Sep 16 19:56 access.log.1
-rw-r–r– 1 root 5.5K Sep 16 19:56 error.log

Note that it said …
log /usr/local/openresty/nginx/logs/access.log.61.zst
doesn’t exist — won’t try to dispose of it

… instead of error renaming … Read-only file system (as logrotate.service does)
Why does it work when running manually as root, but fails when it is executed by the logrotate.service?
I did not make any changes to logrotate.service. For completeness, here is the unit file:
$ systemctl cat logrotate.service
# /lib/systemd/system/logrotate.service
[Unit]
Description=Rotate log files
Documentation=man:logrotate(8) man:logrotate.conf(5)
ConditionACPower=true

[Service]
Type=oneshot
ExecStart=/usr/sbin/logrotate /etc/logrotate.conf

# performance options
Nice=19
IOSchedulingClass=best-effort
IOSchedulingPriority=7

# hardening options
# details: https://www.freedesktop.org/software/systemd/man/systemd.exec.html
# no ProtectHome for userdir logs
# no PrivateNetwork for mail deliviery
# no ProtectKernelTunables for working SELinux with systemd older than 235
# no MemoryDenyWriteExecute for gzip on i686
PrivateDevices=true
PrivateTmp=true
ProtectControlGroups=true
ProtectKernelModules=true
ProtectSystem=full
RestrictRealtime=true

Now I’m running out of options. Any help in troubleshooting the problem is much appreciated.

X ITM Cloud News

Marisa

Leave a Reply

Next Post

Ubuntu installer crashes with grub-install /dev/sda failed fatal error

Wed Sep 16 , 2020
Spread the love          I’m trying to install Ubuntu 20.04.1 in a machine for dual boot with Windows 7, via a bootable flash drive. I’ve made the memory partitions for swap, esp, home etc. But, the installation failed for some reason the first time. And every time I try to install overwriting […]
X- ITM

Cloud Computing – Consultancy – Development – Hosting – APIs – Legacy Systems

X-ITM Technology helps our customers across the entire enterprise technology stack with differentiated industry solutions. We modernize IT, optimize data architectures, and make everything secure, scalable and orchestrated across public, private and hybrid clouds.

This image has an empty alt attribute; its file name is x-itmdc.jpg

The enterprise technology stack includes ITO; Cloud and Security Services; Applications and Industry IP; Data, Analytics and Engineering Services; and Advisory.

Watch an animation of  X-ITM‘s Enterprise Technology Stack

We combine years of experience running mission-critical systems with the latest digital innovations to deliver better business outcomes and new levels of performance, competitiveness and experiences for our customers and their stakeholders.

X-ITM invests in three key drivers of growth: People, Customers and Operational Execution.

The company’s global scale, talent and innovation platforms serve 6,000 private and public-sector clients in 70 countries.

X-ITM’s extensive partner network helps drive collaboration and leverage technology independence. The company has established more than 200 industry-leading global Partner Network relationships, including 15 strategic partners: Amazon Web Services, AT&T, Dell Technologies, Google Cloud, HCL, HP, HPE, IBM, Micro Focus, Microsoft, Oracle, PwC, SAP, ServiceNow and VMware

.

X ITM