If you have a running process, one that you want to keep running, on one of your servers that needs to be supervised, then Daemontools is, in my opinion, the perfect choice. It can keep a process running, keep logs on those processes, and just simply give you one less thing to monitor on your server. Most will prefer to use upstart, which is fine, but I think it sucks compared to Daemontools.

What is Daemontools?

daemontools is a collection of tools for managing UNIX services. supervise monitors a service. It starts the service and restarts the service if it dies. Setting up a new service is easy: all supervise needs is a directory with a run script that runs the service.

multilog saves error messages to one or more logs. It optionally timestamps each line and, for each log, includes or excludes lines matching specified patterns. It automatically rotates logs to limit the amount of disk space used. If the disk fills up, it pauses and tries again, without losing any data.

Why should you use Daemontools?

Observe this link.

Why is /service better than inittab, ttys, init.d and rc.local? Why should my package rely on svscan and supervise?

Answer: Several reasons:

inittab ttys init.d (upstart) rc.local /service
Easy service installation and removal No No Yes No Yes
Easy first-time service startup No No No No Yes
Reliable restarts Yes Yes No No Yes
Easy, reliable signalling No No No No Yes
Clean process state Yes Yes No No Yes
Portability No No No No Yes

Easy service installation and removal.

With /service and init.d, creating your new service means linking some files into a centralized directory, and removing the service means removing those files. This is easy for you to automate.

In contrast, with inittab, ttys, and rc.local, creating a new service means adding some lines to a centralized file, and removing the service means locating and removing those lines. This is much more difficult to automate; there are no portable tools for editing the file. Most packages leave it to the user to do the editing.

Easy first-time service startup.

With /service, once you've created your service, it automatically starts within five seconds.

In contrast, with inittab, ttys, init.d, and rc.local, starting the service means an extra command.

Reliable restarts.

With /service, inittab, and ttys, if your daemon dies, it is automatically restarted.

In contrast, with init.d and rc.local, daemons are not monitored. Your daemon will fail to start if, for example, a previously started daemon has temporarily chewed up almost all available memory. The system administrator has to manually restart your daemon after he notices the problem.

Easy, reliable signalling.

With /service, the system administrator can use svc to control your daemon. For example:

  • svc -h /service/yourdaemon: sends HUP
  • svc -t /service/yourdaemon: sends TERM, and automatically restarts the daemon after it dies
  • svc -d /service/yourdaemon: sends TERM, and leaves the service down
  • svc -u /service/yourdaemon: brings the service back up
  • svc -o /service/yourdaemon: runs the service once

In contrast, with inittab,ttys, init.d , and rc.local, you have to go to extra work to locate the daemon process ID if you want to send it signals. This is not easy to do reliably.

Clean process state.

With /service, inittab, and ttys, when the system administrator restarts a service, the service receives the same fresh process state that it received at boot time.

In contrast, with init.d and rc.local, you have to go to tons of extra work to clean up environment variables, resource limits, controlling ttys, etc. Programmers screw this up all the time, even when they don't care about portability to systems with different process states; system administrators then encounter mysterious failures when they restart daemons.

Portability.

With /service, your program works the same way on every system: Linux, BSD, Solaris, etc.

In contrast, inittab, ttys, init.d and rc.local vary from system to system. Some systems don't have init.d, for example, and the ones that do have it don't agree on its location. This is extremely annoying for cross-platform system administrators.

Installing Daemontools

NOTE: Everything needs to be ran as root

mkdir -p /package
chmod 1755 /package
cd /package

wget http://cr.yp.to/daemontools/daemontools-0.76.tar.gz
tar -xpf daemontools-0.76.tar.gz
rm -f daemontools-0.76.tar.gz
cd admin/daemontools-0.76
package/install

IF INSTALL FAILS

ed ./src/conf-cc
1s/$/ -include errno.h/
wq

Starting Daemontools

apt-get install csh "(for Ubuntu/Debian users)"
csh -cf '/command/svscanboot &'

Starting Daemontools on Boot

sed -i "1 a\csh -cf '/command/svscanboot &'" /etc/rc.local
chmod +x /etc/rc.local

Making Services

I personally like putting my services in the /services directory as daemontools will look for a new service in the /service directory. It makes it easier for tab completion. :-)

mkdir /services

mkdir /services/somerandomservice
echo -e '#!/bin/sh\nexec somerandomcommand' > /services/somerandomservice/run
chmod 755 /services/somerandomservice/run
ln -s /services/somerandomservice/ /service/

Linking your service's daemon directory to your /service directory will tell daemontools to bring your service up. You can check the uptime of your service with svstat /service/somerandomservice

(note: if your service does not keep a longer uptime than 1 second, then something is wrong and you need to check your run script, see debugging for more info)

Run Service With a Specific User

To do this is no different from making a service. You'll just use the setuidgid tool that ships with daemontools.

Observe:

mkdir /services/somerandomuserservice
ed /services/somerandomuserservice/run
a
#!/bin/sh
exec setuidgid somerandomuser somerandomcommand
.
w
!chmod 755 %
q
ln -s /services/somerandomuserservice/ /service/

Now with some Ruby applications, you may experience problems with this method because setuidgid is much more simple from a full login. It only sets the uid and gid as the name implies. If you need the full login functionality then try this:

mkdir /services/somerandomuserservice
ed /services/somerandomuserservice/run
a
#!/bin/sh
exec su - somerandomuser -c 'exec somerandomcommand'
.
w
!chmod 755 %
q
ln -s /services/somerandomuserservice/ /service/

Logging Services

adduser logger

mkdir /services/somerandomservice/log
mkdir /services/somerandomservice/log/main
chown logger /services/somerandomservice/log/main

ed /services/somerandomservice/log/run
a
#!/bin/sh
exec setuidgid logger multilog t ./main
.
w
!chmod 755 %
q

Debugging

I can only help you with debugging daemontools here. Drop in #isotope11 on freenode if you have further problems, and someone will surly help.

Checking the run script

cd /service/somerandomservice
svc -d .
./run

Take note of any errors that are preventing your run script from running. Do they ring any bells? Have you Googled the error?

After you've fixed your bug, bring your service back up with:

svc -u /service/somerandomservice

Checking the logs

cd /service/somerandomservice
tail -n5 log/main/current

If your processes is throwing any errors in the system, they will be logged here. If current isn't there, then it has not had an error yet and you can celebrate.

Restart a service

svc -t /service/somerandomservice

Removing Services

rm /service/somerandomservice
svc -dx /services/somerandomservice

David Chapman is a Ruby on Rails developer at isotope|eleven with over 5 years development experience. He's currently majoring in Electrical Engineering with a minor in Computer Science at the University of Alabama at Birmingham. David is a Linux geek, a math geek, a physics geek, a serious gamer, and a reader of Reddit. His super power is eating chips and salsa.