Aaron Toponce

As many of you are probably already aware, I'm a Linux trainer for Guru Labs. We do all Linux training, including Red Hat, Fedora, SUSE, and others. If your company is looking for good, solid Linux training, Gurus at Guru Labs are the cream of the crop, and I'd highly recommend it. I've been doing a lot of Red Hat training lately, and during the training, I cover how to manage services on a Red Hat system. Lately, I've been meaning to translate this over to Ubuntu, and put it in a series of posts. As a result, the subject of this post is all about learning runlevels, and how to manage services effectively, such as Apache, Squid, SSH, and so on, in Ubuntu. So, let's get started.

First off, to understand services, we need to understand runlevels. Runlevels are a way to automatically start and stop services, thus effectively controlling what is running on your box. When you enter a specific runlevel, you are telling your box that you want to stop a specific set of services and/or start a specific set of services. In other words, think of runlevels as categorizing your system. For example, if I enter runlevel 1, I may want to tell my box to go into "single user mode", thus effectively knocking everyone off except for the root user. Also, I may want to stop all services, including networking, taking my box off the network for troubleshooting reasons. Thus, stop Apache, Squid, SSH, and so on, leaving my box in maintenance mode.

On all Linux and Unix systems, runlevels are treated differently. For example, runlevels in Red Hat have different meaning than runlevels in OpenBSD which have a different meaning than Solaris which finally have a different meaning for Ubuntu. Of course, in this post, I'm only gong to worry about Ubuntu runlevels. If you're curious about the runlevels on other systems, there is a great page here dscribing the differences.

So, let's look at the runlevels that affect Ubuntu. They are as follows:

0: Halt / Shut down.  All processes are stopped, all filesystems are unmounted, and all users taken offline.  Power can now be safely removed.
1: Single user mode.  Only root allowed login (requires password).  All filesystems listed in /etc/fstab are mounted.  Limited set of services started.
2-5: Multi-user mode.  All filesystems listed in /etc/fstab mounted, graphical environment (if installed) brought up.  Full standard operational mode.
6: Reboot.  As with runlevel 0, all processes stopped, all filesystems unmounted, all users taken offline, the box rebooted, and taken to the default runlevel.

Ubuntu does have one extra special runlevel that in processed before entering any of the regular runlevels above. It's known as 'sulogin', and it is considered an emergency mode if the system will not boot, or there are other troubles. it is denoted by "S" as it's runlevel characterization. We will not worry about runlevel S in this post.

Each Unix and Linux operating system has a default runlevel in which it will boot into. If init is the daemon manager for your system, then you will find an /etc/inittab file that will specify your default runlevel. However, Ubuntu has replaced init with Upstart, an event-driven daemon manager. As such, there is no /etc/inittab file. However, there is an /etc/event.d/rc-default file, which as probably guessed, specifies your default runlevel. Let's take a look at the file:

Parsing through that file, we see that first, it tests to see if we are running in runlevel S, or 'sulogin'. If not, then we have a series of checks to find our default runlevel. The first check is to see if there is in fact an /etc/inittab file existing. If so, we look for the line that says "id:?:initdefault:", where "?" could be any valid number 0 through 9. If that number exists, it is our default runlevel, otherwise, runlevel 2 will be our default, as none of the checks would pass.

So, as discovered, runlevel 2 is the default runlevel on Ubuntu. Furthermore, runlevels 2 through 5 are all exactly the same by default. Thus, entering runlevel 4 will have no effect on your box if already running in runlevel 2. What's the point of having so many if they're the same then? Well, a few reasons. First, if you followed that link above, showing the different runlevels on different systems, 0 through 6 is not uncommon, and many of the Unices and Linuxes take advantage of every one of them. So, I guess you could say they're present for historical reasons. However, having that many runlevels gives the system administrator the flexibility to manage a specific runlevel at his/her convenience. For example, maybe runlevel 2 should be treated differently than 3 through 5, starting a different set of services than the rest. Maybe I don't want runlevel 2 to start the graphical mode, but keep it text-based only. This could be useful in troubleshooting my graphical display without it running. You get the idea.

Ok, fine. The next question inevitably asked, is how can I tell what runlevel I'm currently in? Simple, just run the 'runlevel' command:

aaron@kratos:~ 1347 % runlevel
N 2

"N" tells me that there was "None" for my previous runlevel. In other words, this box was just booted into my current runlevel, which I can see as the 2nd identifier, "2". This is one thing that you may like or hate about the 'runlevel' command, but it only shows you your current runlevel and the previous runlevel. No more, no less. However, generally, this is sufficient.

As such, the next question you are probably asking yourself, is how do I change runlevels? This is done with the 'telinit' command. The command tells Upstart, our services manager, what services to stop and start when we enter a specific runlevel. For example, if I wanted to reboot my box from the command line, knowing that runlevel 6 is a reboot, I could enter, as root:

aaron@kratos:~ 1348 % sudo telinit 6
Password:

Ok. So, now we've learned what runlevels are, how to check my current running runlevel, and even how to change to a different runlevel. The next concept in this post is to see what runlevels are doing to the system under the hood. In other words, how does the system know that I want to knock everyone off the box except for root in runlevel 1? How does the system know to start the graphical display in runlevels 2 through 5? The answers to these questions lie in simple shell scripts.

Each runlevel has it's own directory, which contains what we call "K-scripts" and "S-scripts". These directories reside in /etc. In fact, let's get back to our terminal, and type the following, noticing the output:

aaron@kratos:~ 1349 % ls -ld /etc/rc*/
drwxr-xr-x 2 root root 4096 2008-02-16 18:06 /etc/rc0.d/
drwxr-xr-x 2 root root 4096 2008-02-16 18:06 /etc/rc1.d/
drwxr-xr-x 2 root root 4096 2008-02-16 18:06 /etc/rc2.d/
drwxr-xr-x 2 root root 4096 2008-02-16 18:06 /etc/rc3.d/
drwxr-xr-x 2 root root 4096 2008-02-16 18:06 /etc/rc4.d/
drwxr-xr-x 2 root root 4096 2008-02-16 18:06 /etc/rc5.d/
drwxr-xr-x 2 root root 4096 2008-02-16 18:06 /etc/rc6.d/
drwxr-xr-x 2 root root 4096 2007-10-15 17:36 /etc/rcS.d/

That's interesting. I have directories that have the same number as runlevels. Could this be a coincidence? Hardly. When I enter "telinit 4" on the command line, for example, I'm telling Upstart to enter /etc/rc4.d, and execute the K-scripts and S-scripts found in that directory in a specific order. Same goes for entering "telinit 2", "telinit 0" or "telinit 6". Every directory having a specific set of shell scripts that it's going to execute. So, let's dig into one of these directories. Let's start with /etc/rc0.d/:

aaron@kratos:~ 1350 % ls /etc/rc0.d/ 
K01gdm           K20postfix                          S15wpa-ifupdown
K01usplash       K25hwclock.sh                       S20sendsigs
K16avahi-daemon  K50alsa-utils                       S30urandom
K16dhcdbd        K59mountoverflowtmp                 S31umountnfs.sh
K18consolekit    K99laptop-mode                      S40umountfs
K19sendmail      README                              S60umountroot
K20apport        S01linux-restricted-modules-common  S90halt

Here, I can see only 20 scripts and a README file listed. I notice further that I have my K-scripts and S-scripts that I've mentioned earlier. In fact, the K-scripts and S-scripts are numbered. This has to do with order in which to execute those scripts. Let's look at them in detail. First, Upstart is going to execute the K-scripts then it will execute the S-scripts. K-scripts stop a service, while S-scripts start a service. So, the first script it will execute when it enters this directory is "K01gdm". This script manages my graphical display. In other words, I'm going to kill my X session before doing anything else. Next, I'll stop usplash followed my avahi-doemon, etc.

Once all the K-scripts are executed, then I execute the S-scripts. Just as with the K-scripts, they are also executed in order. So, the first script I start is "S01-linux-restricted-modules-common". This script manages my restricted drivers for the kernel, returning nothing more than the exit code for /sbin/lrm-manager, and logging the result. Looking further, I have "S15wpa-ifupdown" which is managing my WPA helper library for wireless devices. We notice further that the last script executed in this directory is "S90halt", which shuts the box down fully. Each of those scripts, as mentioned before, are just shell scripts, and can be opened with any text editor. If you're curious about what a script is doing, open it up and read it.

Let's look at runlevel 2 this time:

aaron@kratos:~ 1351 % ls /etc/rc2.d 
README                       S20hotkey-setup   S30gdm
S05vbesave                   S20makedev        S50ntp
S10acpid                     S20nvidia-kernel  S89anacron
S10powernowd.early           S20postfix        S89atd
S10sysklogd                  S20powernowd      S89cron
S10xserver-xorg-input-wacom  S20rsync          S98usplash
S11klogd                     S21sendmail       S99acpi-support
S12dbus                      S22consolekit     S99laptop-mode
S12hal                       S24avahi-daemon   S99rc.local
S19cupsys                    S24dhcdbd         S99rmnologin
S20apport                    S25bluetooth      S99stop-readahead

First off, we may notice that there are no K-scripts in the runlevel 2 directory. In other words, we aren't interested in stopping any services when we enter runlevel 2. Rather, we are only interested in starting services. Second, we may notice a logical order to the scripts. For example, we start power management right out the gate with S10acpid and S10powernowd.early. Next, we start S10sysklogd, which is our generic logging daemon for logging services. It makes sense to start this daemon before starting specific services, as if they fail to start, I will have a log as to the reason. Also notice that anacron is starting before cron (S89anacron to S89cron). This makes sense, as we want to run any cron jobs that weren't run before running them again. The other daemons start in their numerical order as specified.

Looking in that directory is a bit deceptive, however. Although we see our K-scripts and S-scripts in each directory, they are actually not separate scripts. They are one script actually residing in /etc/init.d/. Let's take a look:

aaron@kratos:~ 1352 % ls -l /etc/rc2.d
total 4
-rw-r--r-- 1 root root 556 2007-10-04 05:17 README
lrwxrwxrwx 1 root root  17 2008-01-09 00:45 S05vbesave -> ../init.d/vbesave
lrwxrwxrwx 1 root root  15 2008-01-09 00:45 S10acpid -> ../init.d/acpid
lrwxrwxrwx 1 root root  25 2008-01-09 00:45 S10powernowd.early -> ../init.d/powernowd.early
lrwxrwxrwx 1 root root  18 2008-01-09 00:45 S10sysklogd -> ../init.d/sysklogd
lrwxrwxrwx 1 root root  34 2008-01-09 00:45 S10xserver-xorg-input-wacom -> ../init.d/xserver-xorg-input-wacom
lrwxrwxrwx 1 root root  15 2008-01-09 00:45 S11klogd -> ../init.d/klogd
lrwxrwxrwx 1 root root  14 2008-01-09 00:45 S12dbus -> ../init.d/dbus
lrwxrwxrwx 1 root root  13 2008-01-09 00:45 S12hal -> ../init.d/hal
lrwxrwxrwx 1 root root  16 2008-01-09 00:45 S19cupsys -> ../init.d/cupsys
lrwxrwxrwx 1 root root  16 2008-01-09 00:45 S20apport -> ../init.d/apport
lrwxrwxrwx 1 root root  22 2008-01-09 00:45 S20hotkey-setup -> ../init.d/hotkey-setup
lrwxrwxrwx 1 root root  17 2008-01-09 00:45 S20makedev -> ../init.d/makedev
lrwxrwxrwx 1 root root  23 2008-01-09 00:45 S20nvidia-kernel -> ../init.d/nvidia-kernel
lrwxrwxrwx 1 root root  17 2008-02-16 15:44 S20postfix -> ../init.d/postfix
lrwxrwxrwx 1 root root  19 2008-01-09 00:45 S20powernowd -> ../init.d/powernowd
lrwxrwxrwx 1 root root  15 2008-01-09 00:45 S20rsync -> ../init.d/rsync
lrwxrwxrwx 1 root root  18 2008-02-16 18:06 S21sendmail -> ../init.d/sendmail
lrwxrwxrwx 1 root root  20 2008-01-09 00:45 S22consolekit -> ../init.d/consolekit
lrwxrwxrwx 1 root root  22 2008-01-09 00:45 S24avahi-daemon -> ../init.d/avahi-daemon
lrwxrwxrwx 1 root root  16 2008-01-09 00:45 S24dhcdbd -> ../init.d/dhcdbd
lrwxrwxrwx 1 root root  19 2008-01-09 00:45 S25bluetooth -> ../init.d/bluetooth
lrwxrwxrwx 1 root root  13 2008-01-09 00:45 S30gdm -> ../init.d/gdm
lrwxrwxrwx 1 root root  13 2008-01-12 13:41 S50ntp -> ../init.d/ntp
lrwxrwxrwx 1 root root  17 2008-01-09 00:45 S89anacron -> ../init.d/anacron
lrwxrwxrwx 1 root root  13 2008-01-09 00:45 S89atd -> ../init.d/atd
lrwxrwxrwx 1 root root  14 2008-01-09 00:45 S89cron -> ../init.d/cron
lrwxrwxrwx 1 root root  17 2008-01-09 00:45 S98usplash -> ../init.d/usplash
lrwxrwxrwx 1 root root  22 2008-01-09 00:45 S99acpi-support -> ../init.d/acpi-support
lrwxrwxrwx 1 root root  21 2008-01-09 00:45 S99laptop-mode -> ../init.d/laptop-mode
lrwxrwxrwx 1 root root  18 2008-01-09 00:45 S99rc.local -> ../init.d/rc.local
lrwxrwxrwx 1 root root  19 2008-01-09 00:45 S99rmnologin -> ../init.d/rmnologin
lrwxrwxrwx 1 root root  24 2008-01-09 00:45 S99stop-readahead -> ../init.d/stop-readahead

Ahh. We can see that each script in the /etc/rc2.d/ directory is actually a soft link pointing to a script of the same name in the /etc/init.d/ directory. This is consistent with /etc/rc[0-6].d/ directories as well. Every file in those directories just pointing to a single script residing in /etc/init.d/. This makes it easy to manage runlevels, as I'm sure you could guess. Because there is only one script, I can create symbolic links in each runlevel directory to start and stop that specific service. As mentioned before, if it's a K-script, it will stop the service in /etc/init.d/, if it's an S-script, it will start the service in /etc/init.d/. This would be the same as calling the script directly:

aaron@kratos:~ 1353 % sudo /etc/init.d/sendmail start
Password:
 * Starting Mail Transport Agent (MTA) sendmail                          [ OK ]

The above is exactly the same as S21sendmail in the /etc/rc2.d/ directory. In fact, if you call the script directly, with no arguments, you can see that it contains a list of valid options, where start and stop are the only requirements:

aaron@kratos:~ 1354 % sudo /etc/init.d/sendmail
Password:
Invalid command <>
Usage: /etc/init.d/sendmail <command>
        Where <command> is one of the following
          start|stop|restart|restart-if-running
          reload-if-running|reload|force-reload
          newaliases|hoststat|purgestat|mailstats|mailq|runq|control
          status|debug|clean

As we can see, the sendmail script has start, stop, restart, restart-if-running, reload-if-running, reload, force-reload, newaliases, hoststat, purgestat, mailstats, mailq, runq, control, status, debug and clean as valid arguments. However, we'll talk more about the /etc/init.d/ directory in the next post in this series. For now, suffice it to say that the /etc/init.d/ directory contains all my shell scripts that a bunch of soft links point to from /etc/rc[0-6].d/. Specifying a runlevel to enter just means to either call the /etc/init.d/ script with a start option, or with a stop option, pending on whether or not it is an S-script or K-script respectively.

This is a good stopping point before continuing on to the next part of the series: Controlling Runlevels. Hopefully now, you understand the different runlevels, how to enter a runlevel, and what a specific runlevel is doing to your box. We looked at the various commands for identifying what runlevel the box is running in and how to change runlevels. We looked at some shell scripts, seeing what arguments are available when calling a daemon directly, and we found out how to tell what our default runlevel is. In other words, you should be fairly fluent on identifying runlevels now.

For the second post in this series, we'll talk about how to manage those runlevels, giving us the flexibility to control exactly what is started and stopped when I enter a specific runlevel. We'll pretty much be spending our entire time on one command, running through lots of examples.

I now have some homework for you to familiarize yourself with the concept of runlevels:

1) Spend some time looking through the various /etc/rc[0-6].d/ directories, identifying K-scripts and S-scripts.  Do not make any changes or modify any files.
2) Open up and read the shell scripts found in those directories, and see if you can figure out what is happening with each daemon when that script stops or starts the service.
3) Spend some time changing runlevels with the 'telinit' command, and see how it affects your box.
4) Change your default runlevel by changing the /etc/event.d/rc-default file (MAKE A BACKUP BEFORE CHANGING THE FILE).  Reboot your box, to see if it worked.
5) If you have access to a Fedora or Red Hat-based distribution, see how the runlevels differ, and see if you can do the previous 4 steps with that distro as well.