Running info-beamer in production

You have developed your visualization and are not sure how to run it in production? This post highlights a few recommendations.

While developing a visualization you usually run info-beamer manually from the command line. This obviously doesn't work when you run your code in production.

info-beamer doesn't enforce any particular way to run it. So there are a lot of options available. I've learned a few things while developing info-beamer hosted which might help you as well if you decide to use Raspbian (or any other Raspberry PI distribution) to run your visualization.

You can use this tutorial to autostart other programs reliably on the PI as well.

If you don't want to deal with all the complexity, you might also take a look at info-beamer hosted, which solved this for you.

Process control/monitoring

When running your code in production you have to make sure that it runs reliable. This means that it is started after booting up and stays running. Any kind of process monitoring helps you with that. There are multiple options available. While those work, they are also a bit difficult set up. For that reason I personally prefer daemontools, which is pretty simple once you understand how it's supposed to be used. It is also available out of the box for Raspbian which makes things even easier.

Installing daemontools

Installation is pretty simple. Just run

root@raspberry:~# apt-get install daemontools daemontools-run

This will install the daemontools programs as well as make sure that everything will start automatically after booting. If you have a look at the processes running, you'll see this:

root@raspberry:~# ps fax
  PID TTY      STAT   TIME COMMAND
 [...]
 2099 ?        Ss     0:00 /bin/sh /usr/bin/svscanboot
 2107 ?        S      0:00  \_ svscan /etc/service
 2108 ?        S      0:00  \_ readproctitle service errors: .....................
 [...]
root@raspberry:~#

svscanboot is automatically started by init (have a look at /etc/inittab to see where that happens). svscanboot is a tiny program that just starts svscan and readproctitle. The first one monitors the directory /etc/service for programs to start (more on that in a minute). readproctitle is a very small program that logs errors by changing its process title. While this sounds weird it is also pretty useful, since you don't have to configure any logfiles when starting to use daemontools. To read the last few log lines, just run

root@raspberry:~# cat /proc/`pidof readproctitle`/cmdline

You don't use this kind of logging a lot. It's almost only useful when you mess up while setting up daemontools. Programs you start won't use this logging mechanism, so don't worry.

Setting up a service

It's time to setup a service. A service is a program that should run after the system started and should be running no matter what. Using daemontools for each service that you want to start requires you to create a directory that tells daemontools how to start your program. Let's see an example. First we'll create a new directory /service and a subdirectory info-beamer inside that directory:

root@raspberry:~# mkdir -p /service/info-beamer

Inside this directory you need a new executable called run. We'll use a shell script here. So go ahead and create /service/info-beamer/run with the following content:

#!/bin/sh
exec /path/to/info-beamer /path/to/your/node 2>&1

Then make sure it is actually executable by setting the correct permissions:

root@raspberry:~# chmod 755 /service/info-beamer/run

You can now run the run script manually to see if everything works as expected:

root@raspberry:~# cd /service/info-beamer
root@raspberrypi:/service/info-beamer# ./run
[ info-beamer output... ]

While this is already a usable daemontools directory it doesn't do any logging right now. Let's set up logging using the system syslog. Create a new subdirectory /service/info-beamer/log:

root@raspberry:~# mkdir -p /service/info-beamer/log

The create the shell script /service/info-beamer/log/run with the following content:

#!/bin/sh
export SERVICE=$(basename $(dirname $PWD))
exec logger -t $SERVICE

Then make sure it is also executable by setting the correct permissions:

root@raspberry:~# chmod 755 /service/info-beamer/log/run

When daemontools now runs your program, it will start the previous run and this log/run script and feed the output of the former to the input of the latter.

You can now test and check that logging works. Just do this. Once you started run, you can type text that will end up in the syslog:

root@raspberry:~# cd /service/info-beamer/log
root@raspberrypi:/service/info-beamer/log# ./run
hello logfile
this is another log line
^D
root@raspberrypi:/service/info-beamer/log#

You can then check the log file:

root@raspberrypi:~# tail /var/log/syslog
[... more log lines ...]
Jul  1 16:07:32 raspberrypi info-beamer: hello logfile
Jul  1 16:07:38 raspberrypi info-beamer: this is another log line

Starting a service

It is highly recommended to run those final steps while being connected to your PI using SSH. Otherwise info-beamer will overlay its output over the console/X screen and you won't be able to interact with the system. So go ahead now and connect to your PI using SSH.

Now it's time to hand over the newly created directory to daemontools so it can take care of starting the program as well at its logger. To do that, just create a new symlink in /etc/service that points to your service directory:

root@raspberry:~# cd /etc/service
root@raspberry:/etc/service# ln -s /service/info-beamer .

That's all. svscan will detect that there is a new symlinked directory in /etc/service within 5 seconds and start the run and log/run scripts inside that directory.

In the example above it will start info-beamer.

If that doesn't work, have a look at the system log:

root@raspberry:~# tail -f /var/log/syslog

Your should be able to see your problem there. If that doesn't help, have a look at the readproctitle log:

root@raspberry:~# cat /proc/`pidof readproctitle`/cmdline

Managing services

You can stop and start services using daemontools. To stop a service, just run svc like this:

root@raspberry:~# svc -d /service/info-beamer

This will send a SIGTERM signal to the info-beamer process while will stop info-beamer.

To restart info-beamer again, do

root@raspberry:~# svc -u /service/info-beamer

For more options on svc, have a look at its documentation.

Adding more services

You can repeat the above steps if you have more long-running programs that you want to autostart by default. Just add another directory in /service, add a run and log/run script and finally symlink that directory into /etc/service.

A better run script for info-beamer

The above run script wasn't too complicated. It just started info-beamer and provided a directory. When you run info-beamer in production you might want to use a few more options:

#!/bin/sh
export INFOBEAMER_LOG_LEVEL=3

modprobe bcm2708_wdog
export INFOBEAMER_WATCHDOG=15

exec nice -n -5 ionice -c 1 -n 0 /path/to/info-beamer /path/to/your/node 2>&1

The first export statement makes info-beamer a bit more verbose, so you can debug problems easier. Log level 3 is normally enough for that. If you want to minimize log output, you can set that to the default value of 1.

The next two lines load the hardware watchdog kernel module for the PI and tell info-beamer to use this feature. In this mode info-beamer has to provide a constant update to the Linux kernel at least every 15 seconds. If info-beamer fails to provide this update, the PI will reboot. This is useful if your PI hangs badly and info-beamer can't do its job anymore. Hopefully this never happens but if it happens at least you won't have a black screen since the PI will reboot.

Finally the modified exec line now includes a call to nice and ionice. nice will make sure that the info-beamer process gets a high priority when running. This make it less likely that another program causes info-beamer to stutter. ionice is similar but give info-beamer a higher priority for disk access. So loading images or videos won't have to wait for other processes.

Conclusion

daemontools is pretty awesome once your start using it. It make it easy to have your PI start programs reliably after booting up. You should give it a try.

If you want to use a reliable Linux distribution where you don't have to do all those steps yourself, you might consider looking at info-beamer hosted. Not only is the operating running on the PI build for reliability, but the complete infrastructure around it makes it incredibly easy to setup a visualization and run it with 100% uptime.

Let me know if you have any questions or improvements.