Quick post on installing ZFS as a kernel module, not FUSE, on Debian GNU/Linux. The documents already exist for getting this going, I’m just hoping to spread this to a larger audience, in case you are unaware that it exists.

First, the Lawrence Livermore National Laboratory has been working on porting the native Solaris ZFS source to the Linux kernel as a kernel module. So long as the project remains under contract by the Department of Defense in the United States, I’m confident there will be continuous updates. You can track the progress of that porting at http://zfsonlinux.org.

Now, download the SPL and ZFS sources. I’m running the latest RC, which seems to be quite stable:

$ mkdir ~/src/{spl,zfs}
$ cd ~/src/spl
$ wget http://github.com/downloads/zfsonlinux/spl/spl-0.6.0-rc8.tar.gz
$ cd ~/src/zfs
$ wget http://github.com/downloads/zfsonlinux/zfs/zfs-0.6.0-rc8.tar.gz

At this point, you will need to install the dependencies for SPL, then go ahead and compile and make the necessary .deb files:

$ sudo aptitude install build-essential gawk alien fakeroot linux-headers-$(uname -r)
$ cd ~/src/spl
$ tar -xf spl-0.6.0-rc8.tar.gz
$ cd spl-0.6.0-rc8
$ ./configure
$ make deb
$ sudo dpkg -i *.deb

Now do the same for ZFS:

$ sudo aptitude install zlib1g-dev uuid-dev libblkid-dev libselinux-dev parted lsscsi
$ cd ~/src/zfs
$ tar -xf zfs-0.6.0-rc8.tar.gz
$ cd zfs-0.6.0-rc8
$ ./configure
$ make deb
$ sudo dpkg -i *.deb

If you’re running Ubuntu, which I know most of you are, you can install the packages from the Launchpad PPA https://launchpad.net/~zfs-native.

A word of note: the manpages get installed to /share/man/. I found this troubling. You can modify your $MANPATH variable to include /share/man/man8/, or by creating symlinks, which is the approach I took:

# cd /usr/share/man/man8/
# ln -s /share/man/man8/zdb.8 zdb.8
# ln -s /share/man/man8/zfs.8 zfs.8
# ln -s /share/man/man8/zpool.8 zpool.8

Now, make your zpool, and start playing:

$ sudo zpool create test raidz sdd sde sdf sdg sdh sdi

It is stable enough to run a ZFS root filesystem on a GNU/Linux installation for your workstation as something to play around with. It is copy-on-write, supports compression, deduplication, file atomicity, off-disk caching, encryption, and much more. At this point, unfortunately, I’m convinced that ZFS as a Linux kernel module will become “stable” long before Btrfs will be stable in the mainline kernel. Either way, it doesn’t matter to me. Both are Free Software, and both provide the long needed features we’ve needed with today’s storage needs. Competition is healthy, and I love having choice. Right now, that choice might just be ZFS.

First, let’s look at the basic setup for getting an interface online with DHCP. The file we’ll be looking at this entire time is the /etc/network/interfaces file:

auto eth0
allow-hotplug eth0
iface eth0 inet dhcp

The first line tells the kernel to bring the “eth0″ interface up when the system boots. The second line tells the kernel to start the interface if a “hotplug” event is triggered. The third line defines the configuration of the “eth0″ interface. In this case, it should use IPv4, and should request an IP address from a DHCP server. A static configuration could look like this:

auto eth0
allow-hotplug eth0
iface eth0 inet static
    address 10.19.84.2
    network 10.19.84.0
    gateway 10.19.84.1
    netmask 255.255.255.0

The first two lines remain the same. In the third line, we have decided to use static addressing, rather than dynamic. Then, we followed through by configuring the interface. It’s important to note that the indentation is not required. I only indented it for my benefit.

What about bonding? Simple enough. Suppose you have 2 NICs, one on the motherboard, and other in a PCI slot, and you want to ensure high availability, should the PCI card die. Then you could do something like this:

auto eth0
iface eth0 inet manual
    post-up ifconfig $IFACE up
    pre-down ifconfig $IFACE down

auto eth1
iface eth1 inet manual
    post-up ifconfig $IFACE up
    pre-down ifconfig $IFACE down

auto bond0
iface bond0 inet static
    bond-slaves eth0 eth1
    # LACP configuration
    bond_mode 802.3ad
    bond_miimon 100
    bond_lcap_rate faste
    bond_xmit_hash_policy layer2+3
    address 10.19.84.2
    network 10.19.84.0
    gateway 10.19.84.1
    netmask 255.255.255.0

Technically, I don’t need to tell the kernel to bring up interfaces eth0 and eth1, if I tell the kernel to bring up bond0, and slave the eth0 and eth1 interfaces. But, this configuration illustrates some points. First, there are the pre-up, up, post-up, pre-down, down, and post-down commands that you can use in your network interfaces(5) file. Each does something to the interface at different times during the configuration. Also notice I’m using the $IFACE variable. There are others that exist, that allow you to create scripts for your interfaces. See http://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_scripting_with_the_ifupdown_system for more information.

On the bonded interface, I’m putting in two slaves, then setting some bonding configuration that I want, such as using 802.3ad mode. Of course, the interface is static, so I provided the necessary information.

What if we wanted to add our bonded interface to a VLAN? Simple. Just append a dot “.” and the VLAN number you want the interface in. Like so:

auto bond0
iface bond0 inet manual
    bond-slaves eth0 eth1
    # LACP configuration
    bond_mode 802.3ad
    bond_miimon 100
    bond_lcap_rate faste
    bond_xmit_hash_policy layer2+3

auto bond0.42
iface bond0.42 inet static
    address 10.19.84.2
    network 10.19.84.0
    gateway 10.19.84.1
    netmask 255.255.255.0
    # necessary due to a bonding bug in vlan tools
    vlan-raw-device bond0

Bring the interface up, the verify that the kernel has assigned it to the right VLAN:

$ sudo cat /proc/net/vlan/config
VLAN Dev name    | VLAN ID
Name-Type: VLAN_NAME_TYPE_RAW_PLUS_VID_NO_PAD
bond0.42        | 42  | bond0

Notice that I specified “vlan-raw-device bond0″. This is due to a bonding bug in the VLAN tools, where merely specifying which VLAN the interface should be in by its name is not enough. You must also tell the kernel the bonded interface that the VLAN interface should be in.

How about bridged devices:

auto bond0
iface bond0 inet manual
    bond-slaves eth0 eth1
    # LACP configuration
    bond_mode 802.3ad
    bond_miimon 100
    bond_lcap_rate faste
    bond_xmit_hash_policy layer2+3

auto bond0.42
iface bond0.42 inet manual
    post-up ifconfig $IFACE up
    pre-down ifconfig $IFACE down
    # necessary due to a bonding bug in vlan tools
    vlan-raw-device bond0

auto br42
iface br42 inet static
    bridge_ports bond0.42
    address 10.19.84.1
    netmask 255.255.255.0
    network 10.19.84.0
    gateway 10.19.84.1

The only new thing here is the “bridge_ports” command. In this case, our bridged device is bridging our bond0.42 interface, which is in VLAN 42. Imagine having a KVM or Xen hypervisor that has a guest that needs to be in several VLANs. How would you setup all those bridges? Simple. Just create a VLAN interface for each VLAN, then create a bridge for each bonded interface in that VLAN.

Lastly, what about virtual IPs? I’ve heard that you can assign multiple IP addresses to a single NIC. How do you set that up? Simple. Just add a colon “:” the append a unique number. For example, say I have only one NIC, but wish to have 2 IP addresses, each in different networks:

auto eth0
iface eth0 inet static
    address 10.19.84.2
    netmask 255.255.255.0
    network 10.19.84.0
    gateway 10.19.84.1

auto eth0:1
iface eth0:1 inet static
    address 10.13.37.2
    netmask 255.255.255.0
    network 10.13.37.0

It’s important to note that you generally only need one default gateway to get out. Your kernel will route packets accordingly. If you must specify multiple gateways, then you must manually make edits to the kernel’s routing table, if everything isn’t setup correctly.

Of course, we could combine everything we learned here. See if you can make out what each interface is doing:

auto eth0
iface eth0 inet manual
    pre-up ifconfig $IFACE up
    post-down ifconfig $IFACE down

auto eth1
iface eth1 inet manual
    pre-up ifcanfig $IFACE up
    post-down ifconfig $IFACe down

auto bond0
iface bond0 inet manual
    bond-slaves eth0 eth1 eth2 eth3
    # LACP configuration
    bond_mode 802.3ad
    bond_miimon 100
    bond_lacp_rate faste
    bond_xmit_hash_policy layer2+3

auto bond0.42
iface bond0.42 inet static
    address 10.19.84.2
    netmask 255.255.255.0
    netwark 10.19.84.0
    gateway 10.19.84.1
    # necessary due to a bonding up in vlan tools
    vlan-raw-device bond0

auto bond0.42:1
iface bond0.42:1 inet manual
    pre-up ifconfig $IFACE up
    post-down ifconfig $IFACE down
    # necessary due to a bonding bug in vlan tools
    vlan-raw-device bond0

auto br42
iface br42 inet static
    bridge_ports bond0.42:1
    address 10.13.37.2
    netmask 255.255.255.0
    network 10.13.37.0

Lastly, MTU. There is a lot of misinformation out there about frame size. In my professional experience, setting the MTU to 9000 bytes does not result in improved performance. Not noticeably at least. But it does have an effect on the CPU. Setting a larger frame size can result in much lower CPU usage, both on the switch, and in your box. However, some protocols, such as UDP, might break with a 9k MTU. So, use appropriately. At any event, here is how I generally set my MTU when dealing with multiple interfaces:

auto eth0
iface eth0 inet manual
    pre-up ifconfig $IFACE up
    post-down ifconfig $IFACE down
    mtu 9000

auto eth1
iface eth1 inet manual
    pre-up ifcanfig $IFACE up
    post-down ifconfig $IFACe down
    mtu 9000

auto bond0
iface bond0 inet manual
    bond-slaves eth0 eth1
    # LACP configuration
    bond_mode 802.3ad
    bond_miimon 100
    bond_lacp_rate faste
    bond_xmit_hash_policy layer2+3
    mtu 9000

auto bond0.42
iface bond0.42 inet static
    address 10.19.84.2
    netmask 255.255.255.0
    network 10.19.84.0
    gateway 10.19.84.1
    mtu 9000
    # necessary due to a bug in vlan tools
    vlan-raw-device bond0

auto bond0.43
iface bond0.43 inet static
    address 10.13.37.2
    netmask 255.255.255.0
    network 10.13.37.0
    mtu 1500
    # necessary due to a bug in vlan tools
    vlan-raw-device bond0

Note that I set the MTU to 9000 on all interfaces except for bond0.43, which is 1500. This is perfectly acceptable. In all reality, setting the MTU to 1500 on bond0.43 is just capping what bond0 can really do. But, it is important to set the MTU on each interface, otherwise the frame size of 1500 bytes will get set, and you’ll end up chopping up your packets anyway. You must also set the MTU to 9000 on the switch ports as well, and any other server and interfaces that you want jumbo frames on.

Salt is a powerful remote execution manager that can be used to administer servers in a fast and efficient way.

Salt allows commands to be executed across large groups of servers. This means systems can be easily managed, but data can also be easily gathered. Quick introspection into running systems becomes a reality.

Remote execution is usually used to set up a certain state on a remote system. Salt addresses this problem as well, the salt state system uses salt state files to define the state a server needs to be in.

Between the remote execution system, and state management Salt addresses the backbone of cloud and data center management.

Think Puppet on steroids. And done correctly.

At any event, yes, I’m looking for a Debian Developer to sponsor me getting it into Debian proper. If you are an Ubuntu Developer, and could help sponsor me getting it into Ubuntu as well, that would be awesome!

$ dd if=/dev/scd0 of=cdimage.iso # NO!

Or worse yet:

$ cat /dev/scd0 > cdimage.iso # NO!

As you are probably thinking, the problem with the two commands above, is that they provide no error checking while building the image. So, in order to make sure you have all the bits, you need to use another tool, such as using the MD5 hashing algorithm:

$ md5sum /dev/scd0 cdimage.iso
d642d524dd2187834a418710001bbf82  /dev/cdrom
d642d524dd2187834a418710001bbf82  cdimage.iso

Thankfully, the hashes above match. But, what if they didn’t? Then, you get to redo your dd(1) command (or, shudder, cat(1)) from above, and then rerun md5sum(1) to make sure you got all the bits. Doesn’t sound like much fun to me. Thankfully, there is a better way, one which handles the checksum while doing the copy.

You want to use readom(1) (“read optical media”) from the wodim(1) (“write optical disk media”) package. Consider the following command:

$ readom dev=/dev/scd0 f=cdimage.iso # YES!

If readom(1) fails to get the bits during the copy, it will let you know that it’s struggling. If it got all the bits, you know you have them all, because of the error checking during the copy. Sure will save you a lot of time running manual hashes when finished.

Now, what about burning a copy of the ISO image? Surely you use dd(1), yes? Something like:

$ dd if=cdimage.iso of=/dev/scd0 # NO!

NO! Instead, use wodim(1) directly:

$ wodim -v -eject cdimage.iso # YES!

For the same reasons that you want to use readom(1) for creating ISO images from CD/DVD, you want to use wodim(1) for burning ISO images to CD/DVD. What happens if after using dd(1) to create your CD/DVD, the md5sum(1) hash doesn’t line up with the image? You didn’t get all the bits, and created a coaster. Use wodim(1) and should it succeed, you can rest assured that you have all the bits.

So, remember, readom(1) and wodim(1) are the tools you want when creating and/or burning ISO images from the command line. Any other tool, and you’re likely doing it wrong.

The command which(1) (which is often a csh script, although sometimes a compiled binary) is not reliable for this purpose. which(1) may not set a useful exit code, and it may not even write errors to stderr. Therefore, in order to have a prayer of successfully using it, one must parse its output (wherever that output may be written).

Note that which(1)’s output when a command is not found is not consistent across platforms. On HP-UX 10.20, for example, it prints “no qwerty in /path /path /path …”; on OpenBSD 4.1, it prints “qwerty: Command not found.”; on Debian (3.1 through 5.0 at least) and SuSE, it prints nothing at all; on Red Hat 5.2, it prints “which: no qwerty in (/path:/path:…)”; on Red Hat 6.2, it writes the same message, but on standard error instead of standard output; and on Gentoo, it writes something on stderr.

(Quotation and manpage reference additions mine). So, if which(1) is bad news, then what is the “proper” way to determine if a command is in your $PATH? Well POSIX has an answer, and not surprisingly, the command to use is “command”:

The “command” built-in also returns true for shell built-ins. If you absolutely must check only PATH, the only POSIX way is to iterate over it:

There are also Bash built-ins that can be used, should you have Bash installed on your system:

Or:

If you prefer the ZSH (my addition not present in the wiki), as I do, then you can look in the $commands associative array:

I like that at the end of the FAQ, he gives a shell script for using which(1) should it be absolutely necessary. Not only do you have to test for exit code, but you also have to test for common strings in the output, seeing as though which(1) doesn’t always use exit codes properly:

CONCLUSION:
You have many options to find whether or not a command exists in your $PATH, some POSIX, some proper built-ins. Regardless, you should be able to build platform-independent scripts using the proper tools, and using which(1) is not the right tool for the job. Hopefully, this has convinced you of that.

Our family has been tasked with drawing a picture for each letter of the alphabet in an alphabet book for my soon-to-be-born niece. The letter ‘d’ was available, so it was obvious to me what should be drawn. I know when she starts flipping through the pages of the book, she will love this page the best.

1. The command line interface.
2. Various shells, including their script syntax.
3. Builtin programming language support for many languages.
4. Common Unix utilities, such as grep, rsync, ssh, lsof, and others.
5. All the supported filesystems (ZFS, Ext4, Reiser, UFS, etc.).
6. Overall rock-solid stability and reliability.
7. Lack of viruses, trojans, and other malware.
8. Tremendous networking capabilities (PPoE, TCP/IP, etc.).
9. Bulletproof firewall software.
10. Overall builtin security in general (MACs, PAM, etc.).
11. Quality user/group management.
12. System resource usage.
13. Both vertical and horizontal scaling.
14. Portability.
15. Plain text configuration files.
16. Open source kernel and user-space software.
17. Based on standards (POSIX, FHS, LSB, etc.).
18. Vast selection of software choices (various text editors, MUAs, etc.)
19. Simplicity in software design- do one thing, and do it well.
20. Mind-blowing hardware support.
21. Support for hundreds of languages and locales out of the box.

shred -n 3 -v /dev/sda

It works well enough. However, it doesn’t display a real useful progress meter, other than how far it’s done in the wipe, thus leaving it up to you to figure out the speed, while filling up your back scroll in the process. There must be a better way.

I used to “leave my mark” (much like a dog marks a fire hydrant), however, this is quite slow. There are other methods, such as using /dev/urandom, but the entropy from urandom relies on SHA1. While fast, it’s not the speed demon that is AES or other algorithms. There’s /dev/zero, but how do I get random bits from zeros? And more importantly, does it push the drive to it’s bandwidth threshold? Of course, I’ve heard about DBAN, but I’ve had issues with it booting on certain hardware. Lastly, I would like to have a good progress meter as the data goes down on the drive.

Here’s a solution that a friend of mine in an IRC channel suggested:

openssl enc -aes128 -k "foo" < /dev/zero | pv -trb > /dev/sda

The great thing with this command is two fold:

1. It’s fast. It pushes the drive to as fast as it can write data.
2. It provides a convenient progress meter with “pv”

Again, I’m shredding drives with pseudorandom data. I’m not too concerned about the security of the bits going down on the platter. Per corporate regulation, I need to do 3 passes, and I’m confident that the bits coming out of the pipe from OpenSSL using AES-128 will be sufficient. So, for doing 3 passes, I can script it easily enough:

for I in 1 2 3; do
    openssl enc -aes128 -k "$I" < /dev/zero | pv -trb > /dev/sda
done

That works. 1 drive down, 24 to go…

If you have various ways you shred your drive, let me know, and I’ll post it below.

So, when installing on a desktop or laptop, I’m slightly annoyed by the default package sets that are installed. I understand why they’re chosen, and I’m definitely not arguing with the decisions made, however, some of the packages just aren’t for me. So, on my wiki, I’ve been documenting what should be installed and removed on every Debian GNU/Linux installation I make. As soon as the install finishes, I’ll login to the system, pull up my wiki, and copy and paste the following commands as root (trying to keep the horizontal scrolling in your browser to a minimum):

# aptitude install abiword apt-file checkconfig chromium-browser clusterssh deborphan flashplugin-nonfree git-core \
gmrun gnumeric htop network-manager openbox openssh-server python-docutils rst2pdf screen tango-icon-theme \
vim vim-gnome xfce4 xfce4-icon-theme xfce4-terminal xscreensaver-data-extra zsh zsh-doc

# aptitude purge dasher gnome-accessibility gnome-accessibility-themes gnome-mag gnome-orca gok nano \
openoffice.org{,-base,-base-core,-calc,-common,-core,-draw,-emailmerge,-evolution,-filter-binfilter,-filter-mobiledev} \
openoffice.org{-gcj,-gnome,-gtk,-help-en-us,-impress,-java-common,-math,-officebean,-style-galaxy,-style-tango,-thesaurus-en-us,-writer} \
xserver-xorg-input-{all,synaptics,wacom} xserver-xorg-video-{all,apm,ark,ati,chips,cirrus,fbdev,i128,i740,intel,mach64,mga,neomagic} \
xserver-xorg-video-{noveau,nv,openchrome,r128,radeon,radeonhd,rendition,s3,s3virge,savage,siliconmotion,sis,sisusb,tdfx,trident,tseng,v4l,voodoo}

First, I will identify which xserver-xorg-video driver I’ll need for the hardware I’m installing to, and remove that from the purge, even though the VESA driver always works just fine for me, as I’m certainly no gamer, and it’s never let me down in my desire to power any video card at any resolution and frequency I’ve encountered. I’ll also leave the Synaptics driver in place when installing to a laptop.

Second, I realize that this isn’t for everybody. Most people won’t care about having OpenOffice.org or the GNOME accessibility packages installed. I don’t need them, they free up hard drive space for me, and when running updates, downloads are much faster. So, this works for me.

Lastly, I have a hard time deciding between using GNOME and Openbox for my default desktop. I like the GDM for logging in, and I like the power management features that GNOME brings to the table. However, I like the minimalist approach to Openbox, and its configuration capabilities. I login to both from time-to-time, just in case you noticed that I’m installing Openbox, but also removing GNOME packages that would only come from a default GNOME install.

I’ve gotten tired of executing this on every install, and wondered if there was something more I could do. So, I mentioned my frustration on Identi.ca, not really looking for a reply, but one came through at any event (thread here). The reply was to use DPKG to get a list of the software that is installed when I’m finished with my install and purge, then use that list during the next install.

So, how to do this? Well, after you have your system installed the way you want, with all the packages installed and purged to suit your needs, run the following command:

$ dpkg --get-selections \* > packages.txt

You now have a text file with all the packages that are installed on your system. So, when doing a fresh Debian install at a later date, it’s trivial to get these packages installed, so I don’t have to do the install and purge copy/paste that I was doing before. After installing only the base, and nothing more, login to the system, copy the packages.txt file you created via SCP or some other method to the filesystem, and run the following commands (as root):

# apt-get update
# dpkg --set-selections < packages.txt
# apt-get -u dselect-upgrade

You now have your system installed with exactly the packages you want installed, and nothing more. Not only that, but the latest version as well. No need to update after you login (Windows, you could learn something here).

To me, this is beautiful. This is a simplistic way to clone a Debian system, without using utilities like Norton Ghost or CloneZilla, either though both have their place in administration. I just love the simplicity and elegance of this. To me, this makes administration fun. When I can solve simple problems with core system tools, I'm a happy admin.

Long live Debian.

There are still a few features screen includes that tmux omits:

- builtin serial and telnet support; this is bloat and is unlikely to be added to tmux;
- wider platform support, for example IRIX and HP-UX, and for odd terminals.

That’s unfortunate. I’ve found that when developers call a requested feature “bloat”, it’s usually because they don’t use it themselves. That may be the case here. I don’t know. However, I do know that GNU screen supports both serial and telnet connections, and it’s a valued feature for our team.

So, here’s how you can use GNU screen to act as a terminal emulator to a serial null modem connection. Just start a screen session on the correct serial device:

$ screen /dev/ttyS0

If the connection is too slow, and your serial port can handle faster baud rates, then you can set that instead:

$ screen /dev/ttyS0 115200

As is common with null modem connections, if you need to send a break, just send C-a B.

That’s it. Rather straight forward. I know that using minicom or HyperTerminal can sometimes be a pain, so using a more modern terminal, complete with telnet, multiuser, locking and splitting support can make all the difference in the world. So, why tmux won’t support this is beyond me, but it sure makes life behind the serial connection just a bit more enjoyable, and a valuable system administration tool.

Before beginning any installation, you should be very familiar with your hardware, so you know what sort of drivers you’ll need for the installation, and if there will be any compatibility issues. Attempting to put Debian on this machine in the past has failed, due to the network driver not shipping with the Lenny kernel. If you have this netbook, the NIC is an Attansic Technology Atheros AR8132/L1c gigabit ethernet adapter. The driver is open source, however, the hardware is so new, that at the time I had purchased the Mini, the driver hadn’t been included in the mainline kernel. The wireless is a Broadcom BCM4132, which means the firmware is not open source, and as a result, not included with the Debian installer. So, at the time, there was no way to get this netbook online with Debian. However, with the release of the 2.6.29 kernel, the Aetheros driver needed was included, and the development snapshot of the installer now ships that kernel, so we’re good to go with a network installation, and getting the computer online.

All the other hardware that I’ve tested, I have tested before with different hardware other than the Mini, and worked out of the box. So, the installation should be rather straightforward, and booting in the new system should be on par with a working system.

So, in order to perform a Debian GNU/Linux installation via USB, you need only a few things. First, you must grab a boot.img.gz file from the development snapshot of the installer for your hardware. Because the HP Mini is x86 32-bit, I grabbed mine here. Now, you also need a CD image file (ISO format) which will contain the necessary software and installation procedures for the install. I prefer to do network installs, so I grabbed a netinst ISO here.

The boot.img.gz file will contain a bootable syslinux kernel and initial ramdisk, which means it will have the drivers necessary for your hardware. Of course, I got mine from a development snapshot, so I could get the Atheros NIC driver from the latest kernel, but if you have older hardware, maybe the stable version of the boot.img.gz would work better for you. You just need to get it from any hd-media directory appropriate for your architecture. The ISO on the other hand contains the base software for installing to disk, the partitioner and other parts of the installer necessary for performing the installation. The boot.img.gz just gets you started.

Now that you have both files, you’ll need a USB thumb drive that is at least 256MB in size, which shouldn’t be a problem these days. Insert the USB drive into a computer with a working Linux operating system, and determine the appropriate device assigned to your newly inserted drive. You can get this information a number of ways. Probably the best way, is to run the following command before you insert the USB drive:

# tail -f -n 0 /var/log/messages

Then, insert the drive. You’ll see output from the kernel as it discovers the hardware and assigns a device to the drive. For me, my output was this:

Feb 21 08:22:28 hermes kernel: [46103.644130] usb 1-7: new high speed USB device using ehci_hcd and address 7
Feb 21 08:22:28 hermes kernel: [46103.789569] usb 1-7: New USB device found, idVendor=13fe, idProduct=1e00
Feb 21 08:22:28 hermes kernel: [46103.789586] usb 1-7: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Feb 21 08:22:28 hermes kernel: [46103.789599] usb 1-7: Product: USB DISK 2.0    
Feb 21 08:22:28 hermes kernel: [46103.789609] usb 1-7: Manufacturer:         
Feb 21 08:22:28 hermes kernel: [46103.789618] usb 1-7: SerialNumber: 077904015F40
Feb 21 08:22:28 hermes kernel: [46103.789974] usb 1-7: configuration #1 chosen from 1 choice
Feb 21 08:22:28 hermes kernel: [46103.790939] scsi5 : SCSI emulation for USB Mass Storage devices
Feb 21 08:22:33 hermes kernel: [46108.838495] scsi 5:0:0:0: Direct-Access              USB DISK 2.0     PMAP PQ: 0 ANSI: 0 CCS
Feb 21 08:22:33 hermes kernel: [46109.101380] sd 5:0:0:0: [sdb] 4030464 512-byte logical blocks: (2.06 GB/1.92 GiB)
Feb 21 08:22:33 hermes kernel: [46109.101984] sd 5:0:0:0: [sdb] Write Protect is off
Feb 21 08:22:33 hermes kernel: [46109.107382]  sdb:
Feb 21 08:22:33 hermes kernel: [46109.174851] sd 5:0:0:0: [sdb] Attached SCSI removable disk

So, in my case, the newly inserted drive is /dev/sdb. So, armed with this information, I can now prepare the USB drive. This next step should be handled with caution. If you type in, whether intentionally or accidentally the wrong device, disasterous consequences may abound. As a friend once told me: “read twice, type once”. Think what you’re doing before you do it. So, at this point, I just need to send the contents of the boot.img.gz file to the new disk. I would not recommend doing it to a partition, but instead doing it to the whole drive. If you inserted your thumb drive, and you noticed in the output that you have a /dev/sdb and /dev/sdb1, then this means you have a partition table outlining a single partition on the drive /dev/sdb. Ignore the partition, work with the drive itself.

Make sure your USB drive is NOT mounted, then type in the following (this next step will remove any existing partitions and data on the drive):

# umount /dev/sdb*
# zcat boot.img.gz > /dev/sdb

This should only take a couple of seconds to finish. At this point, you’ll have a FAT16 formatted USB drive with a syslinux install on the drive. You will now need to mount the drive and copy the ISO image to the mount point.

# mount /dev/sdb /mnt
# cp debian-504-i386-netinst.iso /mnt
# sync
# umount /dev/sdb

At this point, you have a fully prepared USB thumb drive with all the necessary bits in place to perform a USB installation on your netbook, or other hardware. When you boot from the USB stick, you’ll have the familiar Debian installer interface- automated installation, beginner and expert modes and a rescue environment. Because of this, I would recommend keeping the USB stick close at hand, should you need to troubleshoot your installation any time soon.

When you initialize the installation, the installer will look for an ISO file that contains the Debian software. It will start with /dev/sda, and work it’s way device-by-device and partition-by-partition in order, until it finds the ISO file. Because my drive is also recognized as /dev/sdb on my netbook, it only take a couple seconds. After it has found the ISO image, you’re ready to install, just like you would if you had booted off a CD.

That’s it! Rather straightforward, I think. You only need four times really to complete the job:

• You computer
• A USB disk
• A boot.img.gz file
• An ISO containing the Debian software

Good luck on your USB installs!

The idea is simple. On Fedora and SUSE based operating systems, if /etc/cron.allow AND /etc/cron.deny do not exist on the system, then only the super-user can install cron jobs using the crontab command. However, on Debian and Ubuntu, both files are missing, yet everyone on the system can install a cron job. So, the question was: why does Debian and Ubuntu feel the need to be different from everyone else? Why do they need to deviate from standard practice?

Now, for the record, I don’t care if Debian deviates… much. Debian is an operating system. Sometimes, I think those in the Free Software and GNU/Linux world forget that. Operating systems are free to make the changes necessary for their platform as they see fit. Those changes will likely either make users happy and make the operating system popular, like Ubuntu, or they won’t be good changes, and likely will lose users, like, well, Gentoo (sorry guys, but you have seen better days). I’m all for changes that are thought out and that bring obvious or non-obvious benefits. For example, Debian Squeeze moving away from System V Init to Upstart.

So, the question remains: Is Debian deviating with Vixie cron from what would be considered “standard practice”? Well, to start, I pulled up the crontab(1) man page to see what it says regarding the matter. On Debian, this is what I found:

If the /etc/cron.allow file exists, then you must be listed therein in order to be allowed to use this command. If the /etc/cron.allow file does not exist but the /etc/cron.deny file does exist, then you must not be listed in the /etc/cron.deny file in order to use this command. If neither of these files exists, then depending on site-dependent configuration parameters, only the super user will be allowed to use this command, or all users will be able to use this command. For standard Debian systems, all users may use this command.

I pulled up the same man page on Fedora, and this is what I found:

If the cron.allow file exists, then you must be listed therein in order to be allowed to use this command. If the cron.allow file does not exist but the cron.deny file does exist, then you must not be listed in the cron.deny file in order to use this command. If neither of these files exists, only the super user will be allowed to use this command.

Both man pages document exactly what the behavior of crontab is should both /etc/cron.allow and /etc/cron.deny be missing. Further, the crontab(1) man page mentions a site-wide configuration file for this behavior. On Debian, by default, I reached for /etc/default/cron to find this configuration. Nothing in there seemed to lead me to this behavior. Pulling up /etc/sysconfig/crond on Fedora also lacked the information I was looking for. I dug through /etc/pam.d/cron, /etc/crontab, /etc/init/cron, /etc/init.d/cron, /etc/security/access and just about any other possible configuration file that might be related, and came up empty-handed every time.

So, when in doubt, Use the Source Luke. So, I went to the Debian packaging site to grab the cron source. Why there rather than upstream? Because Debian ships the upstream pristine source in one tarball with the Debian-specific patches in another tarball. This way, I can see what is being patched while staring at the source directly. While I was at it, I grabbed the source RPM from Fedora as well. However, I grabbed it from Fedora 8, as it seems Red Hat has forked Vixie cron to “cronie” around Fedora 9, and I wanted to compare apples to apples.

Now, before I dug through the source, I found one bit of information that actually started laying to rest my suspicions. Paul Vixie developed cron for BSD 4.3. So, I would imagine that Vixie cron is still running on BSD systems, and that the default, intended behavior from Paul Vixie himself would be present on the BSDs. Curious, I fired up FreeBSD, and read the crontab(1) man page:

If the allow file exists, then you must be listed therein in order to be allowed to use this command. If the allow file does not exist but the deny file does exist, then you must not be listed in the deny file in order to use this command. If neither of these files exists, then depending on site-dependent configuration parameters, only the super user will be allowed to use this command, or all users will be able to use this command. The format of these files is one username per line, with no leading or trailing whitespace. Lines of other formats will be ignored, and so can be used for comments.

Interesting. Even FreeBSD says that depending on site-wide configuration parameters, either only the super-user will be able to use crontab or everyone. This is the same wording in the Debian man page. Curious, I looked, and sure enough, both the allow and deny files are missing in /var/cron/, and yet everyone on the system can install cron jobs. This is telling me that Debian is not deviating from upstream, and that Red Hat is. However, I have the source, let’s see what that says.

First, I cracked open the Fedora patches to see if the patch was obvious. To be honest, I was a bit overwhelmed by the sheer number of patches Fedora was applying. Most were for PAM and SELinux, however. But, there was a patch for the crontab(1) man page, and there was a patch against crontab itself. After a bit of digging and parsing the C files, it seemed clear to me that Red Hat was patching crontab to only allow root to install a cron job if both the allow and deny files are missing. This patch does not exist in Debian, nor could I find it in FreeBSD.

So, it seemed clear. Debian was in fact not changing the default behavior of cron, but it was Red Hat who was doing the changing. Further, despite what the documentation says, I could find no site-wide configuration file to modify this behavior- even referenced in the source code. The only way to make the change was to change the code before compilation (so maybe we should submit a bug on the man page).

Digging deeper, I learned that there are many cron systems available for GNU/Linux. It appears Arch Linux is shipping dcron by default (Dillon’s cron), Red Hat has forked Vixie cron to cronie, and Debian and Ubuntu both utilize or will utilize Upstart, which will eventually replace cron entirely. It’s my understanding that launchd on Mac OS X has also replaced cron (although I haven’t verified).

Generally, when I got into discussions with various people about Debian or Ubuntu changing this, that or the other for whatever reason, nine times out of ten, it has been my experience that Debian is not the one deviating, but it is the one who is doing the accusing that is deviating. This example with cron has only been one. I’ve had discussions like this many times before. The only real solid example of Debian deviating from standard that I can come up with quickly off the top of my head, is Apache. The /etc/apache2/(sites,modules}-{available,enabled}/ directories are a break from standard. However, I have found that I prefer this configuration to upstream vanilla, as it makes administering specific modules and websites a bit easier to maintain without affecting others. This is a change that is long term beneficial to Debian.

In conclusion, what does this mean? Is Debian better than Fedora/RHEL/CentOS or any other operating system? While I prefer it on my systems, the answer is of course no. But, when breaking from standard practice is called into question, I’m glad Debian sticks as close to upstream as possible. I understand the need for patches where appropriate, but I would prefer as vanilla as possible so I’m not a fish out of water when I need to move to another operating system that is deploying the same technology. At least from that point I’ll be able to see the changes the new system is making. I understand Arch Linux is about as vanilla as you can get, but until they separate the non-free from the free software and GPG sign their packages, I won’t run it.

Debian it is for me, and I’m glad they have the philosophies they do.

First off, let me start by saying that Debian isn’t perfect. No operating system is. However, I find the flexibility of Debian extremely powerful. So powerful, in fact, that Debian can meet the needs of most individuals and situations. While it may not meet the needs of all individuals all the time, I’m confident that it can either meet the needs of all individuals some of the time, or some of the individuals all the time. Let’s take a look.

Installation
First, as mentioned in my previous post, the Debian installer is fantastic. I won’t cover everything here that I already covered in that post, but I will mention a few things. To start, you can download the entire 5-disk DVD set, in addition to a 1-disk DVD update to get you caught up to the latest stable release, and use this set as your software repository, keeping your system completely offline, should you so desire. You could also download 31 CDs, including 5 additional update CDs for the same thing, should you not have a DVD burner at your disposal.

Of course, not everyone is up do downloading 30GB of software, so, should you desire, you could download just the first DVD or CD to do a complete base “default” install. This way, you’ve only downloaded ~5GB if you grabbed the DVD, or ~700MB if you grabbed the CD. Much better than 30GB.

But, Debian doesn’t stop there. Even 700MB might be too much. So, you can download “net installers” which are substantially smaller images. These installers come in two flavors- businesscard and netinst. The businesscard images are designed to be burned on business card CDs, which only hold 50MB total. As a result, these are great to carry in wallets (I do myself) should you be a Debian system administrator. The netinst image is a bit bigger, roughly ~170MB, give or take. The different with these from the business card images is they contain the base software on the ISO, where the business card relies on an external software repository for that.

Aside from ISOs, you can install Debian from a USB drive, PXE or from a local hard disk should you desire. Debian ships expansive documentation covering how to do each of these in detail, so you’re not left stranded.

Releases
Part of what makes Debian GNU/Linux the universal operating system is the architecture itself. The developers of Debian want to reach as many people as possible with the widest array of hardware and software, while not compromising the philosophies in regards to software itself. As such, the developers of Debian have split the software repositories into 6 repositories:

• oldstable: This is the release that was previously the “stable” release. This software is supported for one year by the security team after it has become “oldstable”. If a new stable release happens within that year, then this release will become “oldoldstable” for the remainder of the year, with the new oldstable receiving a new full year of security updates. This is currently aliased to “etch”.
• stable: This is currently aliased to “lenny”. The stable release is the officially supported release by the security team, meaning that security updates and bug fixes are applied in a timely manner.
• testing: This release becomes the test bed for the next stable release. It has filed against it a number of “release critical” bugs. This count must reach as close to zero as humanly possible, while still keeping the idea of a close release at hand before becoming the next stable. Packages enter this release from the “unstable” branch only after a stringent testing criteria. The testing criteria is:
  • It must have been in unstable for 10, 5 or 2 days, depending on the urgency of the upload.
  • It must be compiled and up to date on all architectures it has previously been compiled for in unstable.
  • It must have fewer release-critical bugs than, or the same number as, the version currently in “testing”.
  • All of its dependencies must either be satisfiable by packages already in “testing”, or be satisfiable by the group of packages which are going to be installed at the same time.
  • The operation of installing the package into “testing” must not break any packages currently in “testing”.

  A package which is said to pass 3 of the above criteria is said to be a “valid candidate”. Packages in this release do not get security updates from the security team. This release is currently aliased to “squeeze”. This release is also coined a “rolling release” as there are no release dates, but updates come in on a near daily basis, fixing bugs and preparing for the next stable release.

• unstable: As the release name implies, packages here are not guaranteed to be stable. Packages could break other packages in this release, and regularly do. Security updates are not applied to packages in this release, however, due to the nature of the release, most packages here are bleeding edge with the latest versions. This release is permanently aliased to “sid”. It is also considered a “rolling release” like testing.
• experimental: This release is not indented for installs. It is solely suited for package building, testing and signing. Packages entering this release have just come through the package queue, and are brand new, usually upstream as well. Quite often, packages here are still in development, usually alpha quality. Packages should not be installed from here, as they can be potentially dangerous to your system, even for experienced users.
• volatile: The packages in the stable release sometimes get old out outdated, as the time between releases could be great. This not only includes binaries, but configuration files, libraries, databases and other pieces of software. As such, the volatile release is aimed at keeping things, such as configuration files, more up-to-date. For example, spam blacklists for SMTP servers. It is important for administrators to keep on top of their spam, so keeping up-to-date spam definitions is critical. This release supplies these definitions. Generally, binaries are not included in this release. All package dependencies in this release are satisfiable in the stable release.

Kernels
Aside from the 6 software releases, of which stable, testing and unstable are named after Toy Story characters from the Disney/Pixar movie, Debian GNU/Linux ships 4 kernels as well. This is part of the reason for the name “Debian GNU/Linux” as the name implies that Debian is an operating system that comprises of mostly GNU software with the Linux kernel. However, other kernels and software can be added. As such, the four kernels we have are:

• Debian GNU/Linux
• Debian GNU/kFreeBSD
• Debian GNU/Hurd
• Debian GNU/NetBSD

Debian GNU/kFreeBSD is the furthest developed of the three kernels outside of the Linux kernel mentioned above. Currently, the FreeBSD kernel has landed in the “testing” release, meaning it will be fully supported by the security team for the next “stable” release, codenamed “Squeeze”. Advantages of this bring the ZFS filesystem to the Debian userland, and the PF firewall from OpenBSD. Debian GNU/kFreeBSD will only be supported on two architectures out the gate, namely i386 and amd64. Debian GNU/Hurd and Debian GNU/NetBSD are still under active and heavy development. In fact, the Debian project seems to be doing more for the Hurd kernel than the GNU project itself, as most Hurd developers are also Debian developers.

CPU Architectures
If this isn’t enough, when the Linux kernel initially released, it only supported Intel 386 back in 1991. Fast forward nearly 20 years later, and the Linux kernel supports a massive array of CPU architectures. The Debian project has strived hard to reach as many of them as they can. As such, under the current stable release, Debian GNU/Linux supports 12 CPU architectures, namely:

• Alpha
• AMD64
• ARM
• EABI ARM (“ARMEL”)
• HP PA-RISC
• Intel x86
• Intel IA-64
• MIPS (big endian)
• MIPS (little endian (“MIPSEL”))
• PowerPC
• IBM S/390
• SPARC

There are three additional CPU architectures that are under development, and will probably find their way into a “stable” release. They are:

• Armeb (big endian ARM processors)
• Atmel’s 32-bit RISC
• Hitachi SuperH
• PowerPC64
• Renesas Technology’s 32-bit RISC

Now granted, not all of the software that is available for the Debian operating system is available on every architecture. The Intel processors get the most attention obviously, as they hold the largest market share. But, package support for each architecture is growing, and the heavy hitters in the packages selection are likely already compiled for that architecture, such as Apache, NFS, OpenLDAP, GNOME, etc. NetBSD might be the only other operating system in the world with more hardware support than Debian.

Repositories
Coupled with all this software and hardware that Debian GNU/Linux supports, you can choose your software based on your personal philosophies toward software freedom. The Debain project prides itself in being an operating system that ships Free Software as defined by the GNU project. As such, by default, a Debian operating system will only ship Free Software, leaving the proprietary software out. However, holding true to the universal operating system paradigm, they have made proprietary software available for installation, should you choose to use it. So, they’ve split out their software repositories as follows:

• main: This repository holds the bulk of software installable from Debian. All software in this repository is deemed Free Software as defined by the Debian Free Software Guidelines (see Appendix). This is the only repository enabled by default on a new Debian GNU/Linux install.
• contrib: This repository also contains Free Software, however, it might rely on proprietary counterparts, such as images or media codecs. This repository must be added by the user manually after install.
• non-free: This repository contains only proprietary software, or software licensed such that it does not meet the Deian Free Software Guidelines. This repository must be added by the user manually after install.

Locality
Because the Debian project is a community-driven project run entirely by volunteers in many countries across the world, it also strives to provide package translation for as many languages as possible. Unlike Red Hat, who can say they support 19 languages out the box, Debian has provided package translation, mostly in part, for nearly 250 languages! However, most of these translations are works in progress, and are not considered complete. If you speak one of these languages, feel free to join in on translating packages to get Debian closer to complete in this area.

Conclusion
Outlining the vast array of software and hardware that Debian supports, coupled with the flexible installer, and package translation for hundreds of languages, truly makes Debian the universal operating system. Nevermind the fact that Debian also appeals to a large crowd of users. Everyone from complete “newbs” to the ultimate hardcore hacker can easily fit within the Debian ecosystem.

However, one thing that continues to impress me about the Debian installer is the extreme amount of choices in which to get Debian installed on your system. You can pick any path, ranging from the complete newbie-have-the-installer-choose-everything-for-you to total hacker control over what you want installed, and everything in between. Looking over the installer, here’s a quick list of what I’ve come up with, and how to get Debian installed on your system:

• Text vs Graphical- The Debian installer supports both a text mode and a graphical mode for getting the operating system installed. When booting the installer, you are presented with a menu that allows you to choose which method you want to take.
• Beginner vs Expert- Further, if you want total control over what gets installed on your system, you can choose to take the expert path. This will ask you many more questions on what you want to install and how you want it configured. As a result, the installations takes a bit longer to get through, but if you’ve done it several times, it’s no biggie.
• Local vs Remote- An operating system installer wouldn’t be complete without the ability to do local as well as remote installations. The Debian installer supports setting up both a VNC server and an OpenSSH server for remotely installing the operating system. It also supports “bootstrapping”.
• Manual vs Automatic- Installing the operating system here and there, one at a time is fine for manual installs. However, if you need many installations to take place, or you want the exact same install to go down on many machines, then you can do an automated install using preseed. There are other ways to do automatic installs, such as Kickstart, Kickseed and FAI.
• Installation vs Rescue mode- Let’s not forget that you’re not installing Debian all the time. The Debian installer supports a rescue mode which will mount any filesystem on your local computer, and give you the ability to troubleshoot why your computer is in the trenches, and how to get it out. Windows, Mac, GNU/Linux, etc. If you can talk to the filesystem, you can rescue the computer.

So, now you’ve booted the installer. You’ve loaded the kernel and you’re ready to start an install. Most experienced Debian users will choose to do a network install. This means that you have access to a server acting as a Debian software repository, from which you’ll pull down the packages. Of course, if you don’t have access to a software repository, you can download all the CD images or DVD images, and do an install completely disconnected from any network. Once the installer is ready to go, you have a variety of options on getting Debian installed on your system. First, let’s look at different ways on getting the installer booted:

• PXE- The Debian installer can be loaded through network booting via PXE using TFTP. If you have a TFTP server, and probably DHCP and DNS as well, setup, you can make installs rather painless using this preferred method.
• CD/DVD- This is probably the most “tried and true” method for getting Debian onto your system. Downloading and burning bootable CDs or DVDs are a great way to get Debian installed, even if using optical media is the slowest method of doing so.
• USB- I personally love this option, as I don’t have to waste CDs. I can create a bootable USB drive by downloading, uncompressing and copying over a boot.img.gz to the drive. Then, I mount the drive, and copy over an ISO image I want to use for the installer, and use this newly created bootable drive to install Debian.
• Local Hard Disk- Lastly, you can start the Debian installer by booting from a partition on a local disk to your system. You just grab a Linux kernel and initial RAM disk, as well as an ISO image, copy it to the front of the disk, make some configuration changes, and reboot. This method is completely host operating system independent.

Once the installer is up and running, you now have a slew of options on how to get access to the software for the install. This is where I think the openSUSE installer might have the upper hand, as it supports installing from a SMB share on a Windows network. However, your options are far from limited:

• HTTP- Accessing a Debian software repository via HTTP is the preferred method, especially if the repository is local to your network. And setting up an HTTP software repository is rather trivial if you have the software to do an install.
• FTP- Of course, you can do the same thing with FTP as you can with HTTP. It’s rather trivial to get software of an FTP repository for the install.
• NFS- If you have an NFS server, you can export the repository over the network, and do an installation over NFS.
• CD/DVD- As already mentioned, you can do a complete offline install by using the CDs or DVDs. This is a very slow method for accessing the software packages, but it is rather trivial.
• ISO- As already mentioned, you can use ISO images for the software source. These can be placed on a CDROM or on an external USB drive. Setting either of these up is slightly different than just burning an image to disk, but it’s still rather trivial, and doesn’t take much time. Plus, it’s fast, and light.

Once the installer has booted, the kernel has been loaded, and other configuration parameters setup, the flexibility of the installer doesn’t stop here. You can install Debian on a RAID array using software RAID. You can setup LVM. You have full encryption support, with even determining the type of encryption you want to support (AES, Twofish, Blowfish, etc) and the key strength. When the drives are setup, with partitions, LVM or RAID, you now have the option to install software. You can choose to just do a “base install” which installs only the bare minimum for a bootable operating system. You can install necessary system tools, a desktop environment, with or without laptop support, and so forth. You can choose to have root login or not by using sudo. You have access to two TTYs during the install, from which you can add many users, setup groups, do additional configuration, and so forth before rebooting into your new install.

The options seem to be virtually endless! I was a Red Hat and SUSE trainer for a bit, and I really grew to love the Anaconda an openSUSE installers. They are powerful, flexible installers. However, after learning what was possible with Debian, it seemed clear to me that the Debian installer held the upper hand. Not because I prefer Debian for my default operating system on all my computers, but because of what was immensely possible with it.

If you are a Debian system administrator, either personally or professionally, I would recommend spending some time with the installer to get a feel for what you can do with it. I think you’ll find that it’s rather impressive, keeping up very well with the “enterprise” solutions that exist out there. Also, spending some time on Google will show you a vast array of documentation on how to use the Debian installer to its fullest. This document might be a good start for you.