Getting notified of failed cron jobs

Question

I'm concerned that cron jobs can fail silently for an indefinite period of time on vanilla Ubuntu Desktop 12.04.1 (Precise), and no one will notice. I would like to get a notification whenever a system cron job prints some output or just fails.

I know it is possible to install a mail server (e.g. postfix), configure it for local-only delivery, set up an alias so that root's mail is delivered to my normal user account and configure a mail client to check my local mailbox.

Are there any lightweight alternatives to this solution in Ubuntu?

score 3 · Accepted Answer · edited Jun 12 '20 at 14:37

You could redirect the error output of your cronjob command to a file. Here is an example of a line in /etc/crontab:

01 3    * * *   user    /bin/command 2>> /var/log/some.file

Then at least you got a clue if errors occured. You might even write a script to notify you over notify-osd or similar tools when the file changes.

Edit:

The file /var/log/syslog reports messages from cron as well. You might want to take a look at that. To get a dedicated log-file for the cron deamon edit /etc/rsyslog.d/50-default.conf and uncomment/edit the line that says:

#cron.*             /var/log/cron.log

Don't know what you'll find there but worth the try. Report how it went.

score 1 · Answer 2 · answered Nov 08 '18 at 19:33

Jenkins is a fairly helpful tool for this sort of thing. I know people tend to think of it as a CI, but it's really just a job execution tool. It will capture the output, run time and exit code status of a job which it uses to determine whether or not the job is a failure.

You can setup ssh-agent to have Jenkins connect to a remote machine, schedule jobs to run just like crons, automatically space them based on run time (instead of "3am"), chain dependencies, rotate logs of output on a per-job basis and integrate with a bunch of outside systems (Slack, Hipchat, etc) via plugins to be notified of failures.

Last time I set this up, it was a huge help. We knew immediately if there were issues anywhere and were able to centrally control and track crons from multiple different systems.

score 1 · Answer 3 · answered Oct 01 '13 at 18:35

"I would like to get a notification whenever a system cron job prints some output or just fails."

I'd recommend using some sort of cron monitoring tool. There are a few out there but I currently use Dead Man's Snitch (https://deadmanssnitch.com) and like it. It will alert you when a cron job doesn't check in. Like you're doing, just curl your unique snitch URL after the job and hit the url. There are a few others out there like probyapp but they aren't free...good luck.

score 0 · Answer 4 · answered Nov 08 '18 at 20:43

On modern Ubuntu systems use

journalctl -u cron.service -x

To notify an error without sending emails, try this:

0 1,13 * * * /usr/bin/php /var/www/html/script.php > /tmp/script.log 2>&1 ; [ $? -ne 0 ] && notify-send -u critical -t 3000 "Cron script.php Failed in Execution" "Type journalctl -u cron.service -x to see more details"

score 0 · Answer 5 · answered Nov 08 '18 at 21:04

0

cronwrap 1.4

web: https://pypi.org/project/cronwrap/

A cron job wrapper that wraps jobs and enables better error reporting and command timeouts.

Install with:

pip install cronwrap

answered Nov 08 '18 at 21:04

abu_bua

11,313

score 0 · Answer 6 · answered Nov 30 '20 at 20:34

Try this (example shown; discussion here: https://unix.stackexchange.com/questions/265026/rsyncs-read-only-file-system-30-error/613594#613594):

## all as one line (broken here for readability):
0  5  *  *  *  <your_username>  nice -n 19
  rsync -aq --delete /mnt/Music/ /mnt/Music_Backups ; EXIT_STATUS=$?;
  if [ $EXIT_STATUS -ne 0 ]; then
  notify-send -i warning -t 0 "/etc/crontab for rsync'd Music Backups failed"
  --icon=dialog-information ;
  mutt -s "/etc/crontab for rsync'd Music Backups failed" <your_email_address>; fi

score 0 · Answer 7 · answered May 29 '23 at 22:58

To get an email every time a cron job, you have scheduled to run, fails:

Redirect all of your standard output to /dev/null or to some file.
Use the MAILTO environment variable with a list (comma separated) of all the email addresses you want to receive the email notifications.

To avoid using the same value of MAILTO for every task you run, you can just write it in the crontab e.g.:

MAILTO="foo@bar.com"
0 5 * * * /bin/some_script > /dev/null

MAILTO="foo@bar.com" to the top of a crontab will cause any output from the cron job to be emailed. So we redirect the STDOUT to null and if any STDERR messages are present, they will get emailed to you.

SRC

score 0 · Answer 8 · answered Nov 05 '16 at 09:36

I have done something like the follows, but my use case is for a script or executable set in cron :

0 1,13 * * * /usr/bin/php /var/www/html/script.php > /tmp/script.log 2>&1 ; [ $? -ne 0 ] && mail -s "Cron Failure Report `date` " my.email@mail.com <<< "Cron script.php Failed in Execution"

hope it helps!

For system wide application, DMS is an option. Kindly update if you found a way other than this to do it.

score -2 · Answer 9 · answered Aug 24 '12 at 16:20

-2

Have you tried writing script? I'm sure there is an easy way like:

if cron.output = fail then notify-send "sorry bro"

answered Aug 24 '12 at 16:20

zuberuber

2,030

Getting notified of failed cron jobs

9 Answers9

Edit:

cronwrap 1.4