25

I have configured unattended-upgrades to install security packages and notify by mail when it does so.

I have noticed that the installation happens at very random times. I know that the latest versions added a random delay up to 30 minutes starting from the cron.daily execution time.

However the delays I am experiencing are much bigger than that. I see unattended-upgrades executing at 9am, 3pm, 12am... The logs show the same, so it is not just the email delivery that takes longer.

The unattended upgrades task is the first one in cron.daily, meaning that there is no previous task with huge execution times.

Anybody experienced a similar thing?

daniel f.
  • 866

5 Answers5

35

After debugging this I found the solution.

The root cause of this issue resides in the fact that under Ubuntu 16.04 and newer, unattended-upgrades uses systemd - not cron - to schedule the updates with a huge randomized delay:

/lib/systemd/system/apt-daily.timer is configured with

OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h

This means that it will run twice a day, at 6:00 and 18:00, with a random delay of up to 12 hours. As this is not always acceptable for production environments, I had to override these settings.

In order to keep the package config files untouched, I defined my override in /etc/systemd/system/apt-daily.timer.d/override.conf (Update: please read the edit at the bottom of this answer for further information on filename and location, as it seems to be slightly subject to change).

There I set

[Timer]
OnCalendar=
OnCalendar=06:00
RandomizedDelaySec=1h

to have unattended-upgrades run at 6:00 plus a random delay of up to an hour.

Then I simply restarted the timer with systemctl restart apt-daily.timer (eventually need to reload the daemon).

Unattended-updates now runs at predictable times again!

Edit: It would seem as if for Ubuntu 18.04 things changed a bit. The override should now be stored in /etc/systemd/system/apt-daily-upgrade.timer.d/override.conf and look like this:

[Timer]
OnCalendar=*-*-* 6:00
RandomizedDelaySec=1h

@PerlDuck has mentioned a way of creating an override file with the right name and location in a comment below. Instead of manually creating a file, please consider running sudo systemctl edit apt-daily.timer

daniel f.
  • 866
13

The best way to update the unattended upgrade time, as I compiled it from various sources and tested on our system, is to exclusively use the systemctl commands and avoid trying to find the proper files to edit.

The only thing you have to know for sure is the service name, which in our case, is apt-daily-upgrade (if unsure, search for it via $ systemctl | grep apt). When a systemd service has a timer defined, it is referenced as #{service_name}.timer, thus it’s apt-daily-upgrade.timer for us.

As the system configuration should not be edited, we’ll have to override the default timer config in systemd. For this you need to copy-edit some parts of the original config, so let’s show it first:

$ systemctl cat apt-daily-upgrade.timer 
# /lib/systemd/system/apt-daily-upgrade.timer
[Unit]
Description=Daily apt upgrade and clean activities
After=apt-daily.timer

[Timer] OnCalendar=--* 6:00 RandomizedDelaySec=12h Persistent=true

[Install] WantedBy=timers.target

We’ll need to update the OnCalendar and RandomizedDelaySec values in the [Timer] section. Let’s create the override config file via the following command:

$ systemctl edit apt-daily-upgrade.timer

This should open an editor with a blank file and we need to put there the amended [Timer] section, at least:

[Timer]
# Reset the system calendar config first
OnCalendar=
# Set a new calendar timer with a 60 minute threshold
OnCalendar=*-*-* 21:00
RandomizedDelaySec=60m

As you can see, we’ve updated the OnCalendar value to trigger the automatic updates in the evenings, instead of mornings. The blank OnCalendar line above it must be present as this config value is additive, i.e. it may be specified more than once and only setting it to a blank value resets all previous OnCalendar values (the ones from the system config).

Once we save the file, we can verify that systemd knows about it (there’s no need to run systemctl daemon-reload, the edit command does that for us upon leaving the editor) by running systemctl again as above:

$ systemctl cat apt-daily-upgrade.timer

/lib/systemd/system/apt-daily-upgrade.timer

[Unit] Description=Daily apt upgrade and clean activities After=apt-daily.timer

[Timer] OnCalendar=--* 6:00 RandomizedDelaySec=12h Persistent=true

[Install] WantedBy=timers.target

/etc/systemd/system/apt-daily-upgrade.timer.d/override.conf

[Timer]

Reset the system calendar config first

OnCalendar= OnCalendar=--* 21:00 RandomizedDelaySec=60m

Now it shows two configurations, with our custom one overriding the default one. Good!

The final check that it all works as expected can be done via the list-timers command of systemctl:

$ systemctl list-timers
NEXT                        LEFT          LAST                        PASSED       UNIT                         ACTIVATES                     
...
Thu 2020-08-06 21:51:36 UTC 12h left      Thu 2020-08-06 07:10:20 UTC 2h 20min ago apt-daily-upgrade.timer      apt-daily-upgrade.service               
...             

Find the proper line in the output and look at the NEXT column - the value in there should reflect the time of your newly configured unattended upgrade.

LSerni
  • 523
  • 6
  • 9
10

The official debian documentation on https://wiki.debian.org/UnattendedUpgrades currently has a mistake in it that is misleading a lot of people. It claims that you can override the upgrade time by creating a file called

/etc/systemd/system/apt-daily-upgrade.d/override.conf

However the correct path is

/etc/systemd/system/apt-daily-upgrade.timer.d/override.conf
PerlDuck
  • 13,885
8

I tried Daniel's solution but the upgrade still ran at incorrect times. Turned out there are two systemd overrides needed:

Used for downloads

/lib/systemd/system/apt-daily.timer

...that is overridden with:

/etc/systemd/system/apt-daily.timer.d/override.conf

Used for upgrading

/lib/systemd/system/apt-daily-upgrade.timer

...that is overridden with:

/etc/systemd/system/apt-daily-upgrade.timer.d/override.conf
0

The apt-daily.timer is set to run twice a day, and will not interrupt a production environment, as it simply updates and downloads the packages that need to be updated. You should only need to override apt-daily-upgrade.timer to a time that would be less likely to impact your production environment as this is when the updates are actually installed. Further, there is a setting to reboot for things that require it in 50unattended-upgrades that you should consider setting so they actually get applied and that also has a timer. Also, as of 22.04 the upgrade timer has been changed to default 6am + 1hr, so most will not need to change this.