1. Child process issues the signal SIGCHLD to the parent.
2. Parent receives SIGCHLD, issues the “wait()” system call.
3. Parent now receives the exit code of the child.
4. Parent reaps the child from the process table.

So, when the child process has finished execution of its task, it will report the exit code to the parent. At this point, the child process will remain in the process table until it receives further instruction from the parent. This wait is the defunct, or zombie state. So, in reality, child processes are in this state all the time. It’s just that normally, the parent process acts on it immediately. When the parent does not respond, then we have the zombie state of that child process.

You can check if there are any zombie processes on your system with the following command:

$ ps -eo pid,ppid,user,args,stat --sort stat

Any state of “Z” is a zombie state. So, the question becomes, how do you clean out the zombie, if it is causing issues with your system? Well, you have 3 options:

1. Physically wait around. Sometimes, the parent is busy, and just hasn’t acknowledged the child. When the parent is free, it could clean it up.
2. Send the “SIGCHLD” signal to the parent process. The above command will give you that output in the “PPID” column.
3. Fully kill the parent process. Any child processes will be orphaned, and picked up by INIT. INIT does frequent reaping of child processes and will reap any zombie states.

Quick post on installing ZFS as a kernel module, not FUSE, on Debian GNU/Linux. The documents already exist for getting this going, I’m just hoping to spread this to a larger audience, in case you are unaware that it exists.

First, the Lawrence Livermore National Laboratory has been working on porting the native Solaris ZFS source to the Linux kernel as a kernel module. So long as the project remains under contract by the Department of Defense in the United States, I’m confident there will be continuous updates. You can track the progress of that porting at http://zfsonlinux.org.

Now, download the SPL and ZFS sources. I’m running the latest RC, which seems to be quite stable:

$ mkdir ~/src/{spl,zfs}
$ cd ~/src/spl
$ wget http://github.com/downloads/zfsonlinux/spl/spl-0.6.0-rc8.tar.gz
$ cd ~/src/zfs
$ wget http://github.com/downloads/zfsonlinux/zfs/zfs-0.6.0-rc8.tar.gz

At this point, you will need to install the dependencies for SPL, then go ahead and compile and make the necessary .deb files:

$ sudo aptitude install build-essential gawk alien fakeroot linux-headers-$(uname -r)
$ cd ~/src/spl
$ tar -xf spl-0.6.0-rc8.tar.gz
$ cd spl-0.6.0-rc8
$ ./configure
$ make deb
$ sudo dpkg -i *.deb

Now do the same for ZFS:

$ sudo aptitude install zlib1g-dev uuid-dev libblkid-dev libselinux-dev parted lsscsi
$ cd ~/src/zfs
$ tar -xf zfs-0.6.0-rc8.tar.gz
$ cd zfs-0.6.0-rc8
$ ./configure
$ make deb
$ sudo dpkg -i *.deb

If you’re running Ubuntu, which I know most of you are, you can install the packages from the Launchpad PPA https://launchpad.net/~zfs-native.

A word of note: the manpages get installed to /share/man/. I found this troubling. You can modify your $MANPATH variable to include /share/man/man8/, or by creating symlinks, which is the approach I took:

# cd /usr/share/man/man8/
# ln -s /share/man/man8/zdb.8 zdb.8
# ln -s /share/man/man8/zfs.8 zfs.8
# ln -s /share/man/man8/zpool.8 zpool.8

Now, make your zpool, and start playing:

$ sudo zpool create test raidz sdd sde sdf sdg sdh sdi

It is stable enough to run a ZFS root filesystem on a GNU/Linux installation for your workstation as something to play around with. It is copy-on-write, supports compression, deduplication, file atomicity, off-disk caching, encryption, and much more. At this point, unfortunately, I’m convinced that ZFS as a Linux kernel module will become “stable” long before Btrfs will be stable in the mainline kernel. Either way, it doesn’t matter to me. Both are Free Software, and both provide the long needed features we’ve needed with today’s storage needs. Competition is healthy, and I love having choice. Right now, that choice might just be ZFS.

However, mounting an image file means knowing where the root filesystem begins, which means knowing how to offset the mount, so you can access your data correctly. I used the following:

First, I setup a loop back device, so I could gather information about its partition setup:

# losetup /dev/loop0 virt01.img
# fdisk -l /dev/loop0

Disk /dev/loop0: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0009bdb7

      Device Boot      Start         End      Blocks   Id  System
/dev/loop0p1        37943296    41940991     1998848   82  Linux swap / Solaris
/dev/loop0p2   *        2048    37943295    18970624   83  Linux

Partition table entries are not in disk order

In this case, the virtual machine filesystem is 21.5 GB in size, in reads and writes in 512 byte blocks. Further, it appears as though swap occupies the second partition, while the ext4 root filesystem occupies the first, and begins at sector 2048, or byte 2048*512=1048576.

So, now I just need to tear down the loop back device, and create it again with an offset of 1048576 bytes, at which point, I should be able to mount the device:

# losetup -d /dev/loop0
# losetup /dev/loop0 virt01.img -o 1048576
# mount /dev/loop0 /mnt
# ls /mnt
bin/   home/            lib32/       mnt/   run/      sys/  vmlinuz@
boot/  initrd.img@      lib64/       opt/   sbin/     tmp/  vmlinuz.old@
dev/   initrd.img.old@  lost+found/  proc/  selinux/  usr/
etc/   lib/             media/       root/  srv/      var/

At this point, I can edit my problematic /mnt/etc/fstab file to fix the troubled boot, and boot it up.

First, let’s look at the basic setup for getting an interface online with DHCP. The file we’ll be looking at this entire time is the /etc/network/interfaces file:

auto eth0
allow-hotplug eth0
iface eth0 inet dhcp

The first line tells the kernel to bring the “eth0″ interface up when the system boots. The second line tells the kernel to start the interface if a “hotplug” event is triggered. The third line defines the configuration of the “eth0″ interface. In this case, it should use IPv4, and should request an IP address from a DHCP server. A static configuration could look like this:

auto eth0
allow-hotplug eth0
iface eth0 inet static
    address 10.19.84.2
    network 10.19.84.0
    gateway 10.19.84.1
    netmask 255.255.255.0

The first two lines remain the same. In the third line, we have decided to use static addressing, rather than dynamic. Then, we followed through by configuring the interface. It’s important to note that the indentation is not required. I only indented it for my benefit.

What about bonding? Simple enough. Suppose you have 2 NICs, one on the motherboard, and other in a PCI slot, and you want to ensure high availability, should the PCI card die. Then you could do something like this:

auto eth0
iface eth0 inet manual
    post-up ifconfig $IFACE up
    pre-down ifconfig $IFACE down

auto eth1
iface eth1 inet manual
    post-up ifconfig $IFACE up
    pre-down ifconfig $IFACE down

auto bond0
iface bond0 inet static
    bond-slaves eth0 eth1
    # LACP configuration
    bond_mode 802.3ad
    bond_miimon 100
    bond_lcap_rate faste
    bond_xmit_hash_policy layer2+3
    address 10.19.84.2
    network 10.19.84.0
    gateway 10.19.84.1
    netmask 255.255.255.0

Technically, I don’t need to tell the kernel to bring up interfaces eth0 and eth1, if I tell the kernel to bring up bond0, and slave the eth0 and eth1 interfaces. But, this configuration illustrates some points. First, there are the pre-up, up, post-up, pre-down, down, and post-down commands that you can use in your network interfaces(5) file. Each does something to the interface at different times during the configuration. Also notice I’m using the $IFACE variable. There are others that exist, that allow you to create scripts for your interfaces. See http://www.debian.org/doc/manuals/debian-reference/ch05.en.html#_scripting_with_the_ifupdown_system for more information.

On the bonded interface, I’m putting in two slaves, then setting some bonding configuration that I want, such as using 802.3ad mode. Of course, the interface is static, so I provided the necessary information.

What if we wanted to add our bonded interface to a VLAN? Simple. Just append a dot “.” and the VLAN number you want the interface in. Like so:

auto bond0
iface bond0 inet manual
    bond-slaves eth0 eth1
    # LACP configuration
    bond_mode 802.3ad
    bond_miimon 100
    bond_lcap_rate faste
    bond_xmit_hash_policy layer2+3

auto bond0.42
iface bond0.42 inet static
    address 10.19.84.2
    network 10.19.84.0
    gateway 10.19.84.1
    netmask 255.255.255.0
    # necessary due to a bonding bug in vlan tools
    vlan-raw-device bond0

Bring the interface up, the verify that the kernel has assigned it to the right VLAN:

$ sudo cat /proc/net/vlan/config
VLAN Dev name    | VLAN ID
Name-Type: VLAN_NAME_TYPE_RAW_PLUS_VID_NO_PAD
bond0.42        | 42  | bond0

Notice that I specified “vlan-raw-device bond0″. This is due to a bonding bug in the VLAN tools, where merely specifying which VLAN the interface should be in by its name is not enough. You must also tell the kernel the bonded interface that the VLAN interface should be in.

How about bridged devices:

auto bond0
iface bond0 inet manual
    bond-slaves eth0 eth1
    # LACP configuration
    bond_mode 802.3ad
    bond_miimon 100
    bond_lcap_rate faste
    bond_xmit_hash_policy layer2+3

auto bond0.42
iface bond0.42 inet manual
    post-up ifconfig $IFACE up
    pre-down ifconfig $IFACE down
    # necessary due to a bonding bug in vlan tools
    vlan-raw-device bond0

auto br42
iface br42 inet static
    bridge_ports bond0.42
    address 10.19.84.1
    netmask 255.255.255.0
    network 10.19.84.0
    gateway 10.19.84.1

The only new thing here is the “bridge_ports” command. In this case, our bridged device is bridging our bond0.42 interface, which is in VLAN 42. Imagine having a KVM or Xen hypervisor that has a guest that needs to be in several VLANs. How would you setup all those bridges? Simple. Just create a VLAN interface for each VLAN, then create a bridge for each bonded interface in that VLAN.

Lastly, what about virtual IPs? I’ve heard that you can assign multiple IP addresses to a single NIC. How do you set that up? Simple. Just add a colon “:” the append a unique number. For example, say I have only one NIC, but wish to have 2 IP addresses, each in different networks:

auto eth0
iface eth0 inet static
    address 10.19.84.2
    netmask 255.255.255.0
    network 10.19.84.0
    gateway 10.19.84.1

auto eth0:1
iface eth0:1 inet static
    address 10.13.37.2
    netmask 255.255.255.0
    network 10.13.37.0

It’s important to note that you generally only need one default gateway to get out. Your kernel will route packets accordingly. If you must specify multiple gateways, then you must manually make edits to the kernel’s routing table, if everything isn’t setup correctly.

Of course, we could combine everything we learned here. See if you can make out what each interface is doing:

auto eth0
iface eth0 inet manual
    pre-up ifconfig $IFACE up
    post-down ifconfig $IFACE down

auto eth1
iface eth1 inet manual
    pre-up ifcanfig $IFACE up
    post-down ifconfig $IFACe down

auto bond0
iface bond0 inet manual
    bond-slaves eth0 eth1 eth2 eth3
    # LACP configuration
    bond_mode 802.3ad
    bond_miimon 100
    bond_lacp_rate faste
    bond_xmit_hash_policy layer2+3

auto bond0.42
iface bond0.42 inet static
    address 10.19.84.2
    netmask 255.255.255.0
    netwark 10.19.84.0
    gateway 10.19.84.1
    # necessary due to a bonding up in vlan tools
    vlan-raw-device bond0

auto bond0.42:1
iface bond0.42:1 inet manual
    pre-up ifconfig $IFACE up
    post-down ifconfig $IFACE down
    # necessary due to a bonding bug in vlan tools
    vlan-raw-device bond0

auto br42
iface br42 inet static
    bridge_ports bond0.42:1
    address 10.13.37.2
    netmask 255.255.255.0
    network 10.13.37.0

Lastly, MTU. There is a lot of misinformation out there about frame size. In my professional experience, setting the MTU to 9000 bytes does not result in improved performance. Not noticeably at least. But it does have an effect on the CPU. Setting a larger frame size can result in much lower CPU usage, both on the switch, and in your box. However, some protocols, such as UDP, might break with a 9k MTU. So, use appropriately. At any event, here is how I generally set my MTU when dealing with multiple interfaces:

auto eth0
iface eth0 inet manual
    pre-up ifconfig $IFACE up
    post-down ifconfig $IFACE down
    mtu 9000

auto eth1
iface eth1 inet manual
    pre-up ifcanfig $IFACE up
    post-down ifconfig $IFACe down
    mtu 9000

auto bond0
iface bond0 inet manual
    bond-slaves eth0 eth1
    # LACP configuration
    bond_mode 802.3ad
    bond_miimon 100
    bond_lacp_rate faste
    bond_xmit_hash_policy layer2+3
    mtu 9000

auto bond0.42
iface bond0.42 inet static
    address 10.19.84.2
    netmask 255.255.255.0
    network 10.19.84.0
    gateway 10.19.84.1
    mtu 9000
    # necessary due to a bug in vlan tools
    vlan-raw-device bond0

auto bond0.43
iface bond0.43 inet static
    address 10.13.37.2
    netmask 255.255.255.0
    network 10.13.37.0
    mtu 1500
    # necessary due to a bug in vlan tools
    vlan-raw-device bond0

Note that I set the MTU to 9000 on all interfaces except for bond0.43, which is 1500. This is perfectly acceptable. In all reality, setting the MTU to 1500 on bond0.43 is just capping what bond0 can really do. But, it is important to set the MTU on each interface, otherwise the frame size of 1500 bytes will get set, and you’ll end up chopping up your packets anyway. You must also set the MTU to 9000 on the switch ports as well, and any other server and interfaces that you want jumbo frames on.

% find /path -exec rm -rf {} \;

While certainly functional, it’s not optimal. If there are thousands of files (as is often the case at my job), this command is slow, slow, slow. The reason being are all the excessive fork() and exec() calls for each pass with rm(1). Instead, you could optimize find(1) by using “-delete”:

% find /path -delete

This is much more optimal, but it has one VERY nasty side effect. If you place “-delete” in the wrong spot in your find(1) command, you could delete all the files listed under “/path” before processing the necessary logic. From the find(1) manual:

Warnings: Don’t forget that the find command line is evaluated as an expression, so putting -delete first will make find try to delete everything below the starting points you specified. When testing a find line that you later intend to use with -delete, you should explicitly specify -depth in order to avoid later surprises. Because -delete implies -depth, you cannot usefully use -prune and -delete together.

One nice benefit of “-delete”, however, is the proper handling of NUL characters in your filename, such as spaces, tabs or the newline character. Thankfully, there is another option, which is not only supported in GNU/Linux, but also in FreeBSD (and perhaps others):

% find /path -print0 | xargs -0 rm -rf

This avoids the excessive fork() and exec() system calls from our first command, and doesn’t have the nasty side effects of “-delete”. Further, because of “-print0″ as a find(1) argument, and “-0″ with xargs(1), we can handle files properly with NUL characters. Time the three commands above, and you’ll see that the last is most optimal.

We can squeeze some extra juice out of the command, though. All we need to do is cd(1) to the directory we wish to operate our find(1) command on:

% cd /path && find . -print0 | xargs -0 rm -rf

Working with removing millions of files (yes, I do actually remove that many, often), I have found this latest find(1) command to be the most optimized in terms of sheer speed. It moves. You may find the same results as I.

FYI.

Salt is a powerful remote execution manager that can be used to administer servers in a fast and efficient way.

Salt allows commands to be executed across large groups of servers. This means systems can be easily managed, but data can also be easily gathered. Quick introspection into running systems becomes a reality.

Remote execution is usually used to set up a certain state on a remote system. Salt addresses this problem as well, the salt state system uses salt state files to define the state a server needs to be in.

Between the remote execution system, and state management Salt addresses the backbone of cloud and data center management.

Think Puppet on steroids. And done correctly.

At any event, yes, I’m looking for a Debian Developer to sponsor me getting it into Debian proper. If you are an Ubuntu Developer, and could help sponsor me getting it into Ubuntu as well, that would be awesome!

$ dd if=/dev/scd0 of=cdimage.iso # NO!

Or worse yet:

$ cat /dev/scd0 > cdimage.iso # NO!

As you are probably thinking, the problem with the two commands above, is that they provide no error checking while building the image. So, in order to make sure you have all the bits, you need to use another tool, such as using the MD5 hashing algorithm:

$ md5sum /dev/scd0 cdimage.iso
d642d524dd2187834a418710001bbf82  /dev/cdrom
d642d524dd2187834a418710001bbf82  cdimage.iso

Thankfully, the hashes above match. But, what if they didn’t? Then, you get to redo your dd(1) command (or, shudder, cat(1)) from above, and then rerun md5sum(1) to make sure you got all the bits. Doesn’t sound like much fun to me. Thankfully, there is a better way, one which handles the checksum while doing the copy.

You want to use readom(1) (“read optical media”) from the wodim(1) (“write optical disk media”) package. Consider the following command:

$ readom dev=/dev/scd0 f=cdimage.iso # YES!

If readom(1) fails to get the bits during the copy, it will let you know that it’s struggling. If it got all the bits, you know you have them all, because of the error checking during the copy. Sure will save you a lot of time running manual hashes when finished.

Now, what about burning a copy of the ISO image? Surely you use dd(1), yes? Something like:

$ dd if=cdimage.iso of=/dev/scd0 # NO!

NO! Instead, use wodim(1) directly:

$ wodim -v -eject cdimage.iso # YES!

For the same reasons that you want to use readom(1) for creating ISO images from CD/DVD, you want to use wodim(1) for burning ISO images to CD/DVD. What happens if after using dd(1) to create your CD/DVD, the md5sum(1) hash doesn’t line up with the image? You didn’t get all the bits, and created a coaster. Use wodim(1) and should it succeed, you can rest assured that you have all the bits.

So, remember, readom(1) and wodim(1) are the tools you want when creating and/or burning ISO images from the command line. Any other tool, and you’re likely doing it wrong.

The command which(1) (which is often a csh script, although sometimes a compiled binary) is not reliable for this purpose. which(1) may not set a useful exit code, and it may not even write errors to stderr. Therefore, in order to have a prayer of successfully using it, one must parse its output (wherever that output may be written).

Note that which(1)’s output when a command is not found is not consistent across platforms. On HP-UX 10.20, for example, it prints “no qwerty in /path /path /path …”; on OpenBSD 4.1, it prints “qwerty: Command not found.”; on Debian (3.1 through 5.0 at least) and SuSE, it prints nothing at all; on Red Hat 5.2, it prints “which: no qwerty in (/path:/path:…)”; on Red Hat 6.2, it writes the same message, but on standard error instead of standard output; and on Gentoo, it writes something on stderr.

(Quotation and manpage reference additions mine). So, if which(1) is bad news, then what is the “proper” way to determine if a command is in your $PATH? Well POSIX has an answer, and not surprisingly, the command to use is “command”:

The “command” built-in also returns true for shell built-ins. If you absolutely must check only PATH, the only POSIX way is to iterate over it:

There are also Bash built-ins that can be used, should you have Bash installed on your system:

Or:

If you prefer the ZSH (my addition not present in the wiki), as I do, then you can look in the $commands associative array:

I like that at the end of the FAQ, he gives a shell script for using which(1) should it be absolutely necessary. Not only do you have to test for exit code, but you also have to test for common strings in the output, seeing as though which(1) doesn’t always use exit codes properly:

CONCLUSION:
You have many options to find whether or not a command exists in your $PATH, some POSIX, some proper built-ins. Regardless, you should be able to build platform-independent scripts using the proper tools, and using which(1) is not the right tool for the job. Hopefully, this has convinced you of that.

According to the ssh_config(5) manual:

     LocalCommand
             Specifies a command to execute on the local machine after suc‐
             cessfully connecting to the server.  The command string extends
             to the end of the line, and is executed with the user's shell.
             The following escape character substitutions will be performed:
             ‘%d’ (local user's home directory), ‘%h’ (remote host name), ‘%l’
             (local host name), ‘%n’ (host name as provided on the command
             line), ‘%p’ (remote port), ‘%r’ (remote user name) or ‘%u’ (local
             user name).

             The command is run synchronously and does not have access to the
             session of the ssh(1) that spawned it.  It should not be used for
             interactive commands.

             This directive is ignored unless PermitLocalCommand has been
             enabled.

As mentioned, the used of LocalCommand executes a local command after successfully connecting to the server. I figured this would be a great way to print something to the terminal, letting me know whether or not my client just connected to a production machine, a QA machine, or a development machine.

I wanted to use colors, to make it obvious. I don’t want to make the same mistake twice, so I want it painfully clear what machine I just went to. As a result, if I go to a development or home machine, use green. If I enter a QA machine, use yellow. If I enter a production, or other serious machine I probably shouldn’t be on, use red. As a result, I can take advantage of the ANSI escape sequences for color. In case you forgot, here are the colors and modes:

Colors
\e[{attr1;…;{attrN}m

Text attributes
0 Reset
1 Bright
2 Dim
4 Underscore
5 Blink
7 Reverse
8 Hidden

Foreground Colors
30 Black
31 Red
32 Green
33 Yellow
34 Blue
35 Magenta
36 Cyan
37 White

Background Colors
40 Black
41 Red
42 Green
43 Yellow
44 Blue
45 Magenta
46 Cyan
47 White

So, if I were about to SSH to a production machine, I probably want to make it as obvious as possible. Thus, I could print to the terminal, in blinking, bold, red text “PRODUCTION”. I could use the following command:

print "\e[1;5;31PRODUCTIONm\e[0;m"

Notice that at the end of the sequence, I’m resetting the text attributes. This is because if you don’t do this, you will keep the text attributes in your terminal, and that may have an affect on how the text is displayed when in your remote SSH connection.

A possible ~/.ssh/config file could look like this:

Here is a screenshot in action (without the blink):

At any event, here is the configuration for the theme:

Obviously, this won’t look that great on a terminal with a white background (or really any color other than black). And here is the screenshot:

As you can clearly see, the active window you’re under is bold white with red parentheses around the window name. The previous window you were in is marked with a dash ‘-’ (by default). An alert in a terminal will change the text to bold yellow, so long as you’re not in that window (as you can see with the “mutt” window). It ties in nicely with the 88_madcows.theme file for Irssi, and the ZSH theme I built.

I’m new to building Tmux hardstatus lines, so if there is something I should be doing differently, let me know.

If you have thought about attending, you should register. You can see my talk, if you sign up just for the enthusiast package. You can attend a sponsored lunch (and other things) if you upgrade.

Also, I’ll be hosting the PGP keysigning party at the conference as well (a follow-up post is coming). My key is in the “strong set”, so by me signing your key, I’ll also be bringing you into the strong set as well, if you’re currently not in it. It’s all about strengthening the Web of Trust.

At any event, see you at the fest!

These are best practices from the client perspective, not the server. Many of these points you will have already been familiar with, but some of them not as much. If you sit down and think about what you are doing as a client when you SSH to a server, some of the implications mentioned here make sense.

Lastly, security is a big topic, and not something that can be flipped on and off with a switch. Security always starts with the user. Unfortunately, increasing your security could mean making usability more of a pain in the rear. However, as an OpenSSH client, I hope the techniques mentioned here won’t decrease your usability too bad, while greatly increasing your overall OpenSSH security. With that said, let’s get started.

0. Use ECC first, then RSA, then DSA with maximum bit strength.
When generating your OpenSSH keys, you should be aware of the cryptographic algorithms of the OpenSSH server that you are accessing, and what you can and cannot use. As of OpenSSH version 5.7, elliptic curve cryptography is a supported algorithm. It is proving to be a very secure, robust and light crypto algorithm, but it also must be supported by both the client and the remote server. This means that both the client and the server must be relatively new installations of OpenSSH- something that RHEL 6 and earlier, Debian GNU/Linux Stable 6 and earlier, Ubuntu 10.10 and earlier, and likely many other stable releases from other GNU/Linux operating systems, do not support. When you can, use elliptic curve cryptography for your SSH keys.

If ECC is not available on either your client or server, then you should choose RSA as your key’s crypto. DSA suffers from a weakness, where if the host does not have a sufficiently strong pseudorandom number generator (PRNG), then the “random k” could become exposed from the public key, allowing the attacker to build the private key from the public. The PRNG supplied by GNU/Linux (the device file “/dev/urandom”) is sufficiently strong, providing good random data from entropy pools- thus, GNU/Linux generally doesn’t suffer from this problem. Other operating systems could.

A great demonstration of DSA’s weakness can be found in this excellent blog post, where the author demonstrates how trivial it is to build the private key when the “random k value” is known. Fortunately, RSA does not suffer from this weakness, and also allows you to build larger keys than DSA.

If neither ECC nor RSA is supported by the client nor server, as is the case with some archaic proprietary SSH implementations, then DSA is your only option. Unfortunately, it is also the default for OpenSSH when generating a key. So, you should always get into the habit of passing the “-t [type]” switch when generating your keys.

In all 3 cases, where possible, you should choose the maximum bit strength that the algorithm allows. This will put a bit of strain on the client’s CPU, but it will also give you the greatest strength when authenticating on the wire. When generating your key, use the “-b [size]” switch to specify the bit strength.

1. Use SSH keys and make your servers require them.
There are three reasons why you would want to use SSH keys:

1. SSH keys with a passphrase provide two-factor authentication.
2. SSH passwords can be read in plain text on the remote server.
3. SSH keys aren’t subject to brute force dictionary attacks, like passwords.

Let’s cover each in detail. First, two-factor authentication. SSH keys use public key cryptography. You are given a private and public key when generated. During generation, you are asked to provide a passphrase for the private key. DO NOT GIVE IT AN EMPTY PASSPHRASE! By providing a passphrase to your key, you have enabled the two-factor authentication- something you have (the key) and something you know (the passphrase). Using SSH keys can be troublesome, however. As a result, take advantage of the SSH agent for your system to cache the passphrase locally on your client. This will increase your usability by only entering your key passphrase once, and using the agent to login to other servers without providing the key passphrase again.

Second is reading the user password in plain text on the remote SSH server. You may not think this is possible, as everything in OpenSSH is encrypted, right? Wrong. The two Achilles Heel’s in the connection are the client and servers themselves, where the decryption is taking place. Other users, specifically those who have superuser access, can exploit this. To demonstrate this, make a connection to a server that you have superuser access on. Then, from the client initiate a connection that would require your server password, but don’t supply it. Go back to the server, and find the process of the connection, and run an strace on the PID. Go back to the client, and type in your password. Watch, very likely in horror, the password be displayed on your console.

Here is an excellent writeup by Joseph Hall on this very issue. The problem lies in whether or not you trust those who have superuser access on the server. You should only trust them enough to do their job on the server, and nothing more. Just because they have superuser access on the server, doesn’t mean they can be trusted with your banking account information. Because people use the same passwords over and over across multiple accounts, it’s likely that the password sniffed out of the connection is the same password for their email account. Or bank.

Third, as should be obvious, SSH keys aren’t subject to brute force dictionary attacks like passwords are. If you control the SSH server, and require that SSH keys are the mode of authentication, then brute force attacks will be in vain, regardless how long they try. If you have a publicly facing SSH server, you’re likely aware of how hard your server gets hit from attackers all over the world. By forcing key authentication, all those attacks are in vain.

2. Don’t use a blank passphrase on your keys.
This should come as no surprise, but needs to be mentioned. Your SSH keys are something that should be guarded carefully, with sufficient paranoia. If for any reason, your SSH client is compromised, your SSH keys are a way in to the remote servers that you have access to. If your keys are not protected by passphrases, then after scouring your shell history, or SSH config for hosts to connect to, they’re in the SSH server with little effort. You must protect your keys with strong passphrases!

Now, I understand that there are some like me, who use SSH as the connection for nightly local backups, at which point the client will be asked for the passphrase of the SSH key. Because you wish this to be an automated process, and likely when you’re in bed, you won’t be available all the time to provide the credentials. So, you create an SSH keypair that is not passphrase protected. In such a scenario, here is what I would do:

1. Only install this key on the backup SSH server, and no where else.
2. Do not install this key into an account that has superuser access.
3. Do not install this key into the superuser account.
4. On the backup server, change the key entry in the authorized_keys file to only allow connections from that client using that key (documented how below).

The first, second and third points are easy enough to configure, but how do you configure the fourth point, and is it even possible? When you copy the key to the authorized_keys file on the SSH server, it could look something like this:

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzoblHIUARNP5Kq12QwUqxB6T7m8TWti4LIFcvOCa...

Change the beginning of that entry to this:

from="10.19.84.10",command="/home/user/bin/validate.sh" ssh-rsa AAAAB3NzaC1y...

This tells the server that the only connections allowed to use this key can come from “10.19.84.10″, and when the connection is made, a local script is ran called “validate.sh”. Here are the contents of that script for me:

Make sure the script is executable by the user account on the SSH server, and test it before committing to it as part of your backup policy. If the client host is behind a dynamic IP address, or some other variable that prevents it from having a static address, you can remove the “from=” part, but leave the “command=” part. The goal is to minimize damage that can be done with that key.

Again, this is all if you need to perform an automated backup with SSH keys, where a passphrase cannot be performed. In almost every other case, your SSH key should be protected by a good, strong, loaded with entropy passphrase. Using one from a passwordcard is probably best.

3. Use a separate key for every client you SSH from.
I admit that when I first started using SSH, this didn’t make much sense to me. The amount of keys that I generated was a lot, and managing them became somewhat of a pain. Then, when becoming system administrator of a large organization, and controlling upwards of 300 servers, managing that many keys become immensely intense.

I soon came to the realization that it wasn’t that big of a deal. It is trivial to copy the key to the remote host (this can be done with the “ssh-copy-id(1)” command). Further, for work SSH keys, while I am in charge of managing those keys, I should only worry about work keys at work, and home keys at home. In other words, separate the management of the keys.

But the point of this is to NOT copy the same client key to multiple clients. Think about it. By copying the key around to multiple clients, this means that the key is on every server you ever access from those clients, even if some of the other clients cannot access that server directly. Thus, if a client is compromised, and the key is stolen, it’s difficult to know which client was compromised, and all servers that have that key in their authorized_keys file, are subject to compromise. If you have a different key for each client, then those keys are only on the servers that the client has direct access to, and should a client become compromised, you know which servers to saefguard, and damage is minimal.

4. Limit the number of clients you SSH from.
If an attacker can compromise your client, then they can get access to your SSH keys, as they are stored on the filesystem. Further, they may be able to install a keylogger to log passwords and passphrases, including those for your key. Once this information is gained, then all the accounts on the servers the key is installed on are now compromised as well. By limiting the number of clients you SSH from, you minimize the exposure of SSH servers to the attacker.

5. Don’t “chain” or “loop” logins, but do “star”.

The point relates to the one above it, namely limiting the number of hosts you SSH FROM. There are a few scenarios on how you can create SSH sessions:

• Chain: As a client, you make a connection with an SSH server. Then, from that server, you make another connection to a different SSH server. Once, twice, three times or more, you’ve created a chain of connections from the client to the final host.
• Loop: As a client, you make a connection with an SSH server. Then, from that server, you make a connection back to the client you started from, thus creating a loop with your connections.
• Star: As a client, one connection is made to an SSH server. If another connection is needed to a different SSH server, this is made from the client. For each connection, the original client makes it.

The reasons why this is a “best practice” might not be obvious. Let’s start first with chaining connections. If one client is compromised in the chain, the other clients in the chain might possibly be compromised using the same method. Thus, all connections in the chain could become compromised. And a loop is nothing more than a specific instance of a chain.

Of course, this isn’t always possible. Maybe a chain is the only way to get to an SSH server on a private VLAN or behind a DMZ. These connections might be necessary (typically called “jump hosts”), but they should be used with caution, and as little as possible. When you are done with your task, rather than idle the SSH connection, terminate it to minimize exposure, should an attack occur.

When using a jump host, install netcat(1) on the jump host and use the ProxyCommand configuration paramater. Something like this would be sufficient:

Host inaccessiblehost1 inaccessiblehost2
   ProxyCommand ssh accessiblehost nc -q0 %h %p

With star connections, you minimize the risk of compromise with each of your connections, but security is maximized, even if extra work is required on your end. For example, if you wish to transfer data from one SSH server to another, this may mean bringing the data to the client, then sending it to the destination server.

6. Consider disabling the SSH server on the client you are connecting from.
This rule shouldn’t apply only to the SSH server, but to any daemon you have running on your machine. If you don’t need a service running, then turn it off. By doing so, you minimize the attack vector and greatly increase your security. OpenBSD prides itself in having only two remote holes in the default install in a heck of a long time, but then it also doesn’t have any services running on a default install either.

So, for SSH, because you followed rule #5, and you aren’t doing “chain” or “loop” connections from your client, then this means that you’re not connecting to your own machine that you started your initial connection from. I understand that this isn’t always possible. I have 300+ SSH servers at work, and I can’t possible turn off the SSH server on a host, just because I’m connecting from it. But then, do I need an SSH server on my virtual laptop? Not likely, and if there are things I need, I can turn it on, do my work, then turn it off.

7. Don’t ignore SSH host key warnings.
When you install an SSH server for the first time, it creates a server public and private keypair. This set of keys is what is used for the encryption and decryption between your client and the SSH server. Thus, all traffic that you are sending to that server, is done because you trust that the SSH server key presented to you, is indeed the key of the server itself. So, what happens when you connect a different time, and your client warns you that the public SSH server key has changed? Should you trust the connection?

Depends. Usually, it’s due to the fact that you either reinstalled the SSH server or reinstalled the operating system. In this case, you can move forward, so long as you know that the key presented is the new key from the server. However, someone could setup a proxy SSH server, for the intent of gaining system passwords. Your client will warn you the key is different, and you should pay attention to the warning.

So, in your SSH client config, you generally should not set the following:

Host *
    StrictHostKeyChecking no

Default is “yes”, and it should stay yes in your config. OpenSSH is verbose enough for your benefit. Try to take advantage of it, rather than silence it, or ignore it. After all, it’s your data.

8. Be wary of using X11 forwarding.
While taking advantage of the ability to run remote X applications locally on your client is nice, it has some very grave drawbacks. For example, it’s common for people to launch a remote browser, such as Firefox. What isn’t realized, is that you’re providing usernames and passwords through the browser, which could be caught on the remote machine from which the app is running (taking advantage of X events, for example).

There is a time and a place for X11 forwarding, and I have benefited by it, but generally, it’s not needed. Set the following in your SSH client config, and only enable it when needed:

Host *
    ForwardX11 no

9. Don’t use agent forwarding.
As with X11 forwarding, you are giving ultimate trust to the remote host when forwarding your local agent. If that connection makes a connection to an account with superuser privileges, then a user on the remote SSH server could hijack your agent by connecting to the socket the SSH server creates.

Also, on the client initiating the connection, the superuser must be trusted, which isn’t always the case. The local superuser can access the agent socket on the client, and attack. Thus, it’s generally not a good idea to enable agent forwarding. Thus, as it is by default, the following should be sent in your client SSH config:

Host *
    ForwardAgent no

At any event, in graphical user interfaces, you’ll see floppy disks that represent saving a file to disk. It’s very, very common, yet I doubt many of the younger generation has ever used one. So, how are they supposed to know what it does?

So, here’s a poll for you to take. You’ll need to take the poll at my site, as it uses Ajax calls to the page, which means it won’t work in your RSS reader, or on any other site. I’m just curious to see what falls out.

1. The command line interface.
2. Various shells, including their script syntax.
3. Builtin programming language support for many languages.
4. Common Unix utilities, such as grep, rsync, ssh, lsof, and others.
5. All the supported filesystems (ZFS, Ext4, Reiser, UFS, etc.).
6. Overall rock-solid stability and reliability.
7. Lack of viruses, trojans, and other malware.
8. Tremendous networking capabilities (PPoE, TCP/IP, etc.).
9. Bulletproof firewall software.
10. Overall builtin security in general (MACs, PAM, etc.).
11. Quality user/group management.
12. System resource usage.
13. Both vertical and horizontal scaling.
14. Portability.
15. Plain text configuration files.
16. Open source kernel and user-space software.
17. Based on standards (POSIX, FHS, LSB, etc.).
18. Vast selection of software choices (various text editors, MUAs, etc.)
19. Simplicity in software design- do one thing, and do it well.
20. Mind-blowing hardware support.
21. Support for hundreds of languages and locales out of the box.

shred -n 3 -v /dev/sda

It works well enough. However, it doesn’t display a real useful progress meter, other than how far it’s done in the wipe, thus leaving it up to you to figure out the speed, while filling up your back scroll in the process. There must be a better way.

I used to “leave my mark” (much like a dog marks a fire hydrant), however, this is quite slow. There are other methods, such as using /dev/urandom, but the entropy from urandom relies on SHA1. While fast, it’s not the speed demon that is AES or other algorithms. There’s /dev/zero, but how do I get random bits from zeros? And more importantly, does it push the drive to it’s bandwidth threshold? Of course, I’ve heard about DBAN, but I’ve had issues with it booting on certain hardware. Lastly, I would like to have a good progress meter as the data goes down on the drive.

Here’s a solution that a friend of mine in an IRC channel suggested:

openssl enc -aes128 -k "foo" < /dev/zero | pv -trb > /dev/sda

The great thing with this command is two fold:

1. It’s fast. It pushes the drive to as fast as it can write data.
2. It provides a convenient progress meter with “pv”

Again, I’m shredding drives with pseudorandom data. I’m not too concerned about the security of the bits going down on the platter. Per corporate regulation, I need to do 3 passes, and I’m confident that the bits coming out of the pipe from OpenSSL using AES-128 will be sufficient. So, for doing 3 passes, I can script it easily enough:

for I in 1 2 3; do
    openssl enc -aes128 -k "$I" < /dev/zero | pv -trb > /dev/sda
done

That works. 1 drive down, 24 to go…

If you have various ways you shred your drive, let me know, and I’ll post it below.