Aaron Toponce

This post comes from Matt Taggart, who put together a document about the best practices for using OpenSSH. A lot of the points brought up in that document rang the bells of common sense, and are so good, it’s worth blogging about in hopes that the points mentioned therein reach as many as possible. I’ve also added a couple extra points that I’ve learned with my experience using OpenSSH.

These are best practices from the client perspective, not the server. Many of these points you will have already been familiar with, but some of them not as much. If you sit down and think about what you are doing as a client when you SSH to a server, some of the implications mentioned here make sense.

Lastly, security is a big topic, and not something that can be flipped on and off with a switch. Security always starts with the user. Unfortunately, increasing your security could mean making usability more of a pain in the rear. However, as an OpenSSH client, I hope the techniques mentioned here won’t decrease your usability too bad, while greatly increasing your overall OpenSSH security. With that said, let’s get started.

0. Use ECC first, then RSA, then DSA with maximum bit strength.
When generating your OpenSSH keys, you should be aware of the cryptographic algorithms of the OpenSSH server that you are accessing, and what you can and cannot use. As of OpenSSH version 5.7, elliptic curve cryptography is a supported algorithm. It is proving to be a very secure, robust and light crypto algorithm, but it also must be supported by both the client and the remote server. This means that both the client and the server must be relatively new installations of OpenSSH- something that RHEL 6 and earlier, Debian GNU/Linux Stable 6 and earlier, Ubuntu 10.10 and earlier, and likely many other stable releases from other GNU/Linux operating systems, do not support. When you can, use elliptic curve cryptography for your SSH keys.

If ECC is not available on either your client or server, then you should choose RSA as your key’s crypto. DSA suffers from a weakness, where if the host does not have a sufficiently strong pseudorandom number generator (PRNG), then the “random k” could become exposed from the public key, allowing the attacker to build the private key from the public. The PRNG supplied by GNU/Linux (the device file “/dev/urandom”) is sufficiently strong, providing good random data from entropy pools- thus, GNU/Linux generally doesn’t suffer from this problem. Other operating systems could.

A great demonstration of DSA’s weakness can be found in this excellent blog post, where the author demonstrates how trivial it is to build the private key when the “random k value” is known. Fortunately, RSA does not suffer from this weakness, and also allows you to build larger keys than DSA.

If neither ECC nor RSA is supported by the client nor server, as is the case with some archaic proprietary SSH implementations, then DSA is your only option. Unfortunately, it is also the default for OpenSSH when generating a key. So, you should always get into the habit of passing the “-t [type]” switch when generating your keys.

In all 3 cases, where possible, you should choose the maximum bit strength that the algorithm allows. This will put a bit of strain on the client’s CPU, but it will also give you the greatest strength when authenticating on the wire. When generating your key, use the “-b [size]” switch to specify the bit strength.

1. Use SSH keys and make your servers require them.
There are three reasons why you would want to use SSH keys:

1. SSH keys with a passphrase provide two-factor authentication.
2. SSH passwords can be read in plain text on the remote server.
3. SSH keys aren’t subject to brute force dictionary attacks, like passwords.

Let’s cover each in detail. First, two-factor authentication. SSH keys use public key cryptography. You are given a private and public key when generated. During generation, you are asked to provide a passphrase for the private key. DO NOT GIVE IT AN EMPTY PASSPHRASE! By providing a passphrase to your key, you have enabled the two-factor authentication- something you have (the key) and something you know (the passphrase). Using SSH keys can be troublesome, however. As a result, take advantage of the SSH agent for your system to cache the passphrase locally on your client. This will increase your usability by only entering your key passphrase once, and using the agent to login to other servers without providing the key passphrase again.

Second is reading the user password in plain text on the remote SSH server. You may not think this is possible, as everything in OpenSSH is encrypted, right? Wrong. The two Achilles Heel’s in the connection are the client and servers themselves, where the decryption is taking place. Other users, specifically those who have superuser access, can exploit this. To demonstrate this, make a connection to a server that you have superuser access on. Then, from the client initiate a connection that would require your server password, but don’t supply it. Go back to the server, and find the process of the connection, and run an strace on the PID. Go back to the client, and type in your password. Watch, very likely in horror, the password be displayed on your console.

Here is an excellent writeup by Joseph Hall on this very issue. The problem lies in whether or not you trust those who have superuser access on the server. You should only trust them enough to do their job on the server, and nothing more. Just because they have superuser access on the server, doesn’t mean they can be trusted with your banking account information. Because people use the same passwords over and over across multiple accounts, it’s likely that the password sniffed out of the connection is the same password for their email account. Or bank.

Third, as should be obvious, SSH keys aren’t subject to brute force dictionary attacks like passwords are. If you control the SSH server, and require that SSH keys are the mode of authentication, then brute force attacks will be in vain, regardless how long they try. If you have a publicly facing SSH server, you’re likely aware of how hard your server gets hit from attackers all over the world. By forcing key authentication, all those attacks are in vain.

2. Don’t use a blank passphrase on your keys.
This should come as no surprise, but needs to be mentioned. Your SSH keys are something that should be guarded carefully, with sufficient paranoia. If for any reason, your SSH client is compromised, your SSH keys are a way in to the remote servers that you have access to. If your keys are not protected by passphrases, then after scouring your shell history, or SSH config for hosts to connect to, they’re in the SSH server with little effort. You must protect your keys with strong passphrases!

Now, I understand that there are some like me, who use SSH as the connection for nightly local backups, at which point the client will be asked for the passphrase of the SSH key. Because you wish this to be an automated process, and likely when you’re in bed, you won’t be available all the time to provide the credentials. So, you create an SSH keypair that is not passphrase protected. In such a scenario, here is what I would do:

1. Only install this key on the backup SSH server, and no where else.
2. Do not install this key into an account that has superuser access.
3. Do not install this key into the superuser account.
4. On the backup server, change the key entry in the authorized_keys file to only allow connections from that client using that key (documented how below).

The first, second and third points are easy enough to configure, but how do you configure the fourth point, and is it even possible? When you copy the key to the authorized_keys file on the SSH server, it could look something like this:

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzoblHIUARNP5Kq12QwUqxB6T7m8TWti4LIFcvOCa...

Change the beginning of that entry to this:

from="10.19.84.10",command="/home/user/bin/validate.sh" ssh-rsa AAAAB3NzaC1y...

This tells the server that the only connections allowed to use this key can come from “10.19.84.10″, and when the connection is made, a local script is ran called “validate.sh”. Here are the contents of that script for me:

Make sure the script is executable by the user account on the SSH server, and test it before committing to it as part of your backup policy. If the client host is behind a dynamic IP address, or some other variable that prevents it from having a static address, you can remove the “from=” part, but leave the “command=” part. The goal is to minimize damage that can be done with that key.

Again, this is all if you need to perform an automated backup with SSH keys, where a passphrase cannot be performed. In almost every other case, your SSH key should be protected by a good, strong, loaded with entropy passphrase. Using one from a passwordcard is probably best.

3. Use a separate key for every client you SSH from.
I admit that when I first started using SSH, this didn’t make much sense to me. The amount of keys that I generated was a lot, and managing them became somewhat of a pain. Then, when becoming system administrator of a large organization, and controlling upwards of 300 servers, managing that many keys become immensely intense.

I soon came to the realization that it wasn’t that big of a deal. It is trivial to copy the key to the remote host (this can be done with the “ssh-copy-id(1)” command). Further, for work SSH keys, while I am in charge of managing those keys, I should only worry about work keys at work, and home keys at home. In other words, separate the management of the keys.

But the point of this is to NOT copy the same client key to multiple clients. Think about it. By copying the key around to multiple clients, this means that the key is on every server you ever access from those clients, even if some of the other clients cannot access that server directly. Thus, if a client is compromised, and the key is stolen, it’s difficult to know which client was compromised, and all servers that have that key in their authorized_keys file, are subject to compromise. If you have a different key for each client, then those keys are only on the servers that the client has direct access to, and should a client become compromised, you know which servers to saefguard, and damage is minimal.

4. Limit the number of clients you SSH from.
If an attacker can compromise your client, then they can get access to your SSH keys, as they are stored on the filesystem. Further, they may be able to install a keylogger to log passwords and passphrases, including those for your key. Once this information is gained, then all the accounts on the servers the key is installed on are now compromised as well. By limiting the number of clients you SSH from, you minimize the exposure of SSH servers to the attacker.

5. Don’t “chain” or “loop” logins, but do “star”.

The point relates to the one above it, namely limiting the number of hosts you SSH FROM. There are a few scenarios on how you can create SSH sessions:

• Chain: As a client, you make a connection with an SSH server. Then, from that server, you make another connection to a different SSH server. Once, twice, three times or more, you’ve created a chain of connections from the client to the final host.
• Loop: As a client, you make a connection with an SSH server. Then, from that server, you make a connection back to the client you started from, thus creating a loop with your connections.
• Star: As a client, one connection is made to an SSH server. If another connection is needed to a different SSH server, this is made from the client. For each connection, the original client makes it.

The reasons why this is a “best practice” might not be obvious. Let’s start first with chaining connections. If one client is compromised in the chain, the other clients in the chain might possibly be compromised using the same method. Thus, all connections in the chain could become compromised. And a loop is nothing more than a specific instance of a chain.

Of course, this isn’t always possible. Maybe a chain is the only way to get to an SSH server on a private VLAN or behind a DMZ. These connections might be necessary (typically called “jump hosts”), but they should be used with caution, and as little as possible. When you are done with your task, rather than idle the SSH connection, terminate it to minimize exposure, should an attack occur.

When using a jump host, install netcat(1) on the jump host and use the ProxyCommand configuration paramater. Something like this would be sufficient:

Host inaccessiblehost1 inaccessiblehost2
   ProxyCommand ssh accessiblehost nc -q0 %h %p

With star connections, you minimize the risk of compromise with each of your connections, but security is maximized, even if extra work is required on your end. For example, if you wish to transfer data from one SSH server to another, this may mean bringing the data to the client, then sending it to the destination server.

6. Consider disabling the SSH server on the client you are connecting from.
This rule shouldn’t apply only to the SSH server, but to any daemon you have running on your machine. If you don’t need a service running, then turn it off. By doing so, you minimize the attack vector and greatly increase your security. OpenBSD prides itself in having only two remote holes in the default install in a heck of a long time, but then it also doesn’t have any services running on a default install either.

So, for SSH, because you followed rule #5, and you aren’t doing “chain” or “loop” connections from your client, then this means that you’re not connecting to your own machine that you started your initial connection from. I understand that this isn’t always possible. I have 300+ SSH servers at work, and I can’t possible turn off the SSH server on a host, just because I’m connecting from it. But then, do I need an SSH server on my virtual laptop? Not likely, and if there are things I need, I can turn it on, do my work, then turn it off.

7. Don’t ignore SSH host key warnings.
When you install an SSH server for the first time, it creates a server public and private keypair. This set of keys is what is used for the encryption and decryption between your client and the SSH server. Thus, all traffic that you are sending to that server, is done because you trust that the SSH server key presented to you, is indeed the key of the server itself. So, what happens when you connect a different time, and your client warns you that the public SSH server key has changed? Should you trust the connection?

Depends. Usually, it’s due to the fact that you either reinstalled the SSH server or reinstalled the operating system. In this case, you can move forward, so long as you know that the key presented is the new key from the server. However, someone could setup a proxy SSH server, for the intent of gaining system passwords. Your client will warn you the key is different, and you should pay attention to the warning.

So, in your SSH client config, you generally should not set the following:

Host *
    StrictHostKeyChecking no

Default is “yes”, and it should stay yes in your config. OpenSSH is verbose enough for your benefit. Try to take advantage of it, rather than silence it, or ignore it. After all, it’s your data.

8. Be wary of using X11 forwarding.
While taking advantage of the ability to run remote X applications locally on your client is nice, it has some very grave drawbacks. For example, it’s common for people to launch a remote browser, such as Firefox. What isn’t realized, is that you’re providing usernames and passwords through the browser, which could be caught on the remote machine from which the app is running (taking advantage of X events, for example).

There is a time and a place for X11 forwarding, and I have benefited by it, but generally, it’s not needed. Set the following in your SSH client config, and only enable it when needed:

Host *
    ForwardX11 no

9. Don’t use agent forwarding.
As with X11 forwarding, you are giving ultimate trust to the remote host when forwarding your local agent. If that connection makes a connection to an account with superuser privileges, then a user on the remote SSH server could hijack your agent by connecting to the socket the SSH server creates.

Also, on the client initiating the connection, the superuser must be trusted, which isn’t always the case. The local superuser can access the agent socket on the client, and attack. Thus, it’s generally not a good idea to enable agent forwarding. Thus, as it is by default, the following should be sent in your client SSH config:

Host *
    ForwardAgent no