37

Following on from this question, I have written a simple upstart service (/etc/init/pms.conf) for my headless Ubuntu Server 11.04 box as follows:

start on filesystem and net-device-up IFACE=eth0
stop on runlevel [016]
respawn

exec /home/administrator/pms-current/PMS.sh

I can start (or stop) this service at will from the command line:

service pms start

And I can see that it is indeed running.

However, when I first boot my machine the service does not start. If I SSH into the box and check the service status I get:

$ service pms status
pms stop/waiting

My question is why is this happening? Why isn't my service starting on boot?

UPDATE 1: unsure whether my service was being started and subsequently dying or just wasn't being start at all, I added the following to PMS.sh:

echo "STARTED" > $STARTLOG

This obviously just gives me something to look for. I tested this by starting the service myself and then checking start.log. I then deleted the start.log and rebooted. It wasn't there after the restart, so it seems as though upstart definitely isn't starting my service. I suppose it could be dying at an earlier point in the process, but that seems rather unlikely given the simplicity of it all.

UPDATE 2: I've just upgraded to 11.10 which includes an upstart upgrade, but this problem still occurs.

UPDATE 3: As requested, I've booted with --debug. The output of cat /var/log/syslog | grep init is too long to place in the question, but you view it here.

UPDATE 4: More logs, this time the upstart conf is included at the top. Run 1 and run 2.

Community
  • 1
Kent Boogaart
  • 1,088

12 Answers12

20

I would recommend increasing the verbosity of the job, e.g. by using pre-start/post-start entries.

pre-start script
  logger "pre-start for myprog"
end script

post-start script
  logger "post-start for myprog"
end script

# and for PMS itself:
script
  logger "just before executing PMS"
  exec /home/administrator/pms-current/PMS.sh
end script

Further information at http://upstart.ubuntu.com/cookbook/

Also have a look at http://upstart.ubuntu.com/wiki/Debugging

swift
  • 3,291
Clausi
  • 5,057
15

What's probably happening here is that pms is starting before your network adapters come up, and probably before even the loopback adapter (lo). Assuming we're talking about PS3 Media Server, it's a networked service and it probably doesn't like starting up with no interfaces available.

Try changing your start on criteria to:

start on filesystem and net-device-up IFACE!=lo

Meaning, start after any "real" network interface is up. However, that might not be ideal, if eth0 is the next interface up, PMS starts, but you really want PMS to use wlan0, that won't do. The service will start but it might not have been able to pick the interface you wanted it to listen on. Assuming you know the interface you're going to stream over and it won't be changing, I would hardcode it into the job, e.g.:

start on filesystem and net-device-up IFACE=wlan0

On Oneiric (11.10), you can use the event static-network-up to wait for all statically configured devices. Which is nice because it allows you to write network-dependent jobs without hardcoding an interface. [Note: by "all statically configured devices", I'm referring to using /etc/network/interfaces instead of NetworkManager. It does not mean static in the sense of static IP vs. DHCP.]

Giacomo1968
  • 262
Mark Russell
  • 7,396
3

Managed to fix similar problem by using start on runlevel instead:

start on runlevel [2345]
Laurynas
  • 131
3

I had the same problem and eventually I solved it simply with:

start on runlevel [2345]

without any net-device-up or started networking stuff

This is the complete upstart script, and it works perfectly:

# MyApp

description     "MyApp"
author          "me"

start on runlevel [2345]
stop on runlevel [016]

respawn

exec /usr/bin/myapp 2>> /var/logs/myapp.log
Daniele B
  • 131
3

From examining your syslog the pms process starts with no errors but then after a short while its goal is changed from start to stop meaning it is killed.

This is slightly strange because you have added the repsawn clause so it should attempt to start again after it is stopped but it never does. So I'm guessing you removed the respawn clause.

Between the pms service starting and stopping only 2 services are started ufw and network-interface (eth0), and 1 is started udev-fallback-graphics.

It seems that you process pms is being started in parallel. Unfortunately the upstart documentation is a little bit hazy on the exact differences between start on ... vanilla and start on starting ... and start on started ....

Try changing your startup stanza to 

start on started networking

or just too

start on net-device-up IFACE=eth0

The log output is slightly strange as the net-device-up event comes much later but pms starts before it.

This should ensure that your process only starts once all networking set up is finished i.e. the job has not only started but finished.

Also do not trust log output completely, early in the boot process logging output to any file does not always work. See the answer in Debugging Upstart

Community
  • 1
Ciaran Liedeman
  • 1,389
1

Is your home directory on NFS? Sometimes root can't access NFS.

For the record, in my little test just now on 12.04:

  • start on started networking and start on network-interface-up INTERFACE=eth0 don't work, but

  • start on started network-interface INTERFACE=eth0 does.

Thanks to http://os4.org/wiki/upstart.html for pointing out that initctl list always shows job networking as stopped.

muru
  • 207,228
user94311
  • 11
  • 1
1

I had a similar "no start" problem when I have realized that my script depended on a file that was in my home, and the home was not accessible because was crypted with the standard ubuntu mechanism ( .Private).

start on local-filesystems event is (probably) emitted before the decryption process is ended.

muru
  • 207,228
alessandro
  • 11
  • 1
1

I came across chkconfig during my RHCSA/CE training:

sudo apt-get install chkconfig
sudo chkconfig pms on

You can check it's Oneiric man page for more details on it's capabilities.

Oxwivi
  • 18,499
1

I've found a solution for this but I don't understand it. If I move PMS out of /home/administrator and into /bin/pms with root as the owner, it all works fine.

If I leave it under /home/administrator/ but make sure root is the owner of everything bar the /home/administrator/ directory itself, it still doesn't work.

If I set administrator as owner of everything and change the pertinent part of my script to:

sudo su administrator -c '/home/administrator/pms-current/PMS.sh'

It still doesn't work.

I suppose for now I'll make a /home/root/ directory and move everything there, though I'd really like to fully understand this.

Oxwivi
  • 18,499
Kent Boogaart
  • 1,088
0

In my case, upstart service depended on the script located in vagrant synced folder. Solved the problem using the following line:

start on vagrant-mounted

More info: http://razius.com/articles/launching-services-after-vagrant-mount/

xuhcc
  • 101
0

Similar to @xuhcc, I came here to find out why my Vagrant Upstart script wasn't running. The following is supposed to work:

start on vagrant-mounted

But doesn't in some builds due to the following bug.

https://github.com/mitchellh/vagrant/issues/6074

The workaround listed in the report worked great for me:

$ cat /etc/init/workaround-vagrant-bug-6074.conf 
# workaround for https://github.com/mitchellh/vagrant/issues/6074
start on filesystem
task

env MOUNTPOINT=/vagrant

script
  until mountpoint -q $MOUNTPOINT; do sleep 1; done
  /sbin/initctl emit --no-wait vagrant-mounted MOUNTPOINT=$MOUNTPOINT
end script

Worked great for me

Ian E
  • 101
0

it worked for me (i need start service after iface up):

start on started networking and net-device-up IFACE=wlan1 
stop on shutdown

respawn
respawn limit 10 10
MSS
  • 200
  • 8