Upstart job problems on CentOS 6.x

I needed to launch a process at boot time, monitor and restart it if it crashed. After researching and considering monit, node.js forever, supervisord etc, I chose Upstart. Not that I knew enough about all of them to make an informed decision mind you. But if the distribution (CentOS and many others) has chosen it as a replacement for the venerable System V then who am I to disagree.

Ok enough about why, let's get to the what. It is a simple what like they all are.

I was fighting this one job which I had initially set up to:


start on runlevel [2345]
stop on runlevel [!2345]

but now I wanted to start only when told to start (so as to delay the start at boot time enough to allow my cloud-init script to make some config changes before bringing up the process/server/daemon using those config changes):


start on my-start-event
stop on my-stop-event

And of course I was going to trigger these at the end of the cloud-init script which made the config changes like so:

initctl emit my-start-event

In theory (from docs) it was all supposed to work but it wasn't.

I had tried:

initctl reload-configuration

and so on but no gives.

In the end it was my eagerness to centralize code - had put job.conf in my project under SVN and symlinked to it from /etc/init/job.conf. A little line in the initctl manual was the key to figuring out the problem:


reload-configuration

     Requests that the init(8) daemon reloads its configuration. This  command  is  generally not necessary since init(8) watches its configuration directories with inotify(7) and automatically reloads in cases of changes.

Well the symlink timestamp does not change when the target content changes!! Ok off with it and will implement a deployment script to take the job.conf from project to where it needs to be.