6

I have a PC behind a NAT which makes a reversed SSH connection to my Digitalocean VPC. I utilise this reversed SSH connection from home to login to my office PC (I am authorised to do so) and copy files and do other important things.

Although not often, I noticed that my office PC restarts (due to power failures etc) and breaks the reversed SSH connection it has made with my VPC. In these kind of cases, I am unable to connect from my home PC to my office PC.

I run the following script to make the reversed connection + dynamic proxy to anonymise my traffic ( As I am not required to share browsing information) generated at the office PC.

autossh -CD 8080 -i digitalOcean -R 8081:localhost:22 root@IPofDigitalOceanPC

There is no way I can run this script again on my office PC upon a restart as I am not physically there. In order to solve this problem I installed the following crontab.

Note: rev.sh file contains the above line. The certificate "digitalOcean" and rev.sh is located in Ubuntu home. Therefore, when I execute ./rev.sh in my Ubuntu terminal I obtain a dynamic proxy and also access to my DigitalOcean server. This method works 100%.

However when I install the crontab in the following method, My ubuntu PC never makes a Dynamic proxy. I can see this because when I check this proxy from Google Chrome, it says proxy is refusing connection.

Here are the cronjobs I tried as root's cronjobs. I also tried these as a normal user, still they didn't work.

@reboot bash /home/user/rev.sh 
@reboot /home/user/rev.sh 
@reboot cd /home/user && ./rev.sh

I then installed a crontab a several minutes before the current time and waited for it to execute.

24 12 * * * bash /home/user/rev.sh
24 12 * * * /home/user/rev.sh
24 12 * * * reboot

These did not execute either.

I also tried 48 15 * * * bash /home/user/rev.sh >> test3 and */1 * * * * reboot -f >> test but the test3 and test have nothing. However the files has been CREATED!! by the crontab!

Please be kind enough to help me spot my mistake. There are many similar questions on this website on my issue. I have referred many answers hence but none of them seemed to help.

Denis
  • 337

1 Answers1

1

A better solution would be to make use of watchdog. watchdog is a daemon which will watch running processes and if they exit will automatically restart them.

Install watchdog on Ubuntu 16.04

sudo apt-get install watchdog

How watchdog works

The watchdog(8) daemon will execute scripts in /etc/watchdog.d with the argument test or repair. (see TEST DIRECTORY section of watchdog(8) man page). Your watchdog script handles those two arguments when it checks to see if a process is running and takes an action to repair it.

You can configure watchdog by modifying /etc/watchdog.conf (See watchdog.conf(5)).

Example watchdog.d script

Take for example /etc/watchdog.d/autossh_script (which has 755 permissions and is owned by root).

Note: you may need to customize $targetuser environment variable in the example script. sam is my username.

#!/bin/bash

targetuser=sam

runTest=false
runRepair=false

case $1 in
  test)
    runTest=true
  ;;
  repair)
    runRepair=true
    repairExitCode=$2
  ;;
  *)
    echo 'Error: script needs to be run by watchdog' 1>&2
    exit 1
  ;;
esac

if ${runTest}; then
  #run a test here which will tell the status of your process
  #the exit code of this script will be the repairExitCode if it is non-zero
  if ! pgrep autossh &> /dev/null; then
    #autossh not running; notify watchdog to repair
    exit 1
  else
    #autossh running; no action necessary
    exit 0
  fi
fi

if ${runRepair}; then
  #take an action to repair the affected item
  #use a case statement on $repairExitCode to handle different failure cases
  su - ${targetuser} -c 'nohup autossh -f -- -NCD 8080 -i digitalOcean -R 8081:localhost:22 root@IPofDigitalOceanPC'
  exit 0
fi
  • I added -N to your ssh command in the example script so it only starts the tunnel but does not attempt to create a login shell.
  • I added -f to autossh so that it runs in the background.
  • The example script uses a simple pgrep pattern. However, you can further narrow the test to a specific user or even use a pattern for the process. See pgrep(1) for how to further customize your test using pgrep.

Configuring watchdog

/etc/watchdog.conf and /etc/defaults/watchdog are places to configure watchdog. See watchdog.conf(5).

One thing to note is user scripts are executed once every second by default. I recommend increasing this to at least 30 seconds unless you have a need for more realtime checking. Adjust the interval setting in watchdog.conf.

Troubleshooting

You may need to create the /etc/watchdog.d directory before your script.

/var/log/watchdog/* contains watchdog related logs and errors. If your script outputs to stdout or stderr then it will be written there. On my system I notice my script executes test or repair roughly once every second. If you use echo in your script it should only be temporary for debugging purposes only. Otherwise, discarding output is recommended except in the case of errors.

If your script is not running at all then check the permissions: ls -l /etc/watchdog.d.

Sam Gleske
  • 380