• I am, by definition, “the intended recipient”.
• All information in the email is mine to do with as I see fit and make such financial profit, political mileage, or good joke as it lends itself to. In particular, I may quote it where I please.
• I may take the contents as representing the views of your company.
• This disclaimer overrides any disclaimer or statement of confidentiality that may be included on your message.

% find /path -exec rm -rf {} \;

While certainly functional, it’s not optimal. If there are thousands of files (as is often the case at my job), this command is slow, slow, slow. The reason being are all the excessive fork() and exec() calls for each pass with rm(1). Instead, you could optimize find(1) by using “-delete”:

% find /path -delete

This is much more optimal, but it has one VERY nasty side effect. If you place “-delete” in the wrong spot in your find(1) command, you could delete all the files listed under “/path” before processing the necessary logic. From the find(1) manual:

Warnings: Don’t forget that the find command line is evaluated as an expression, so putting -delete first will make find try to delete everything below the starting points you specified. When testing a find line that you later intend to use with -delete, you should explicitly specify -depth in order to avoid later surprises. Because -delete implies -depth, you cannot usefully use -prune and -delete together.

One nice benefit of “-delete”, however, is the proper handling of NUL characters in your filename, such as spaces, tabs or the newline character. Thankfully, there is another option, which is not only supported in GNU/Linux, but also in FreeBSD (and perhaps others):

% find /path -print0 | xargs -0 rm -rf

This avoids the excessive fork() and exec() system calls from our first command, and doesn’t have the nasty side effects of “-delete”. Further, because of “-print0″ as a find(1) argument, and “-0″ with xargs(1), we can handle files properly with NUL characters. Time the three commands above, and you’ll see that the last is most optimal.

We can squeeze some extra juice out of the command, though. All we need to do is cd(1) to the directory we wish to operate our find(1) command on:

% cd /path && find . -print0 | xargs -0 rm -rf

Working with removing millions of files (yes, I do actually remove that many, often), I have found this latest find(1) command to be the most optimized in terms of sheer speed. It moves. You may find the same results as I.

FYI.

Salt is a powerful remote execution manager that can be used to administer servers in a fast and efficient way.

Salt allows commands to be executed across large groups of servers. This means systems can be easily managed, but data can also be easily gathered. Quick introspection into running systems becomes a reality.

Remote execution is usually used to set up a certain state on a remote system. Salt addresses this problem as well, the salt state system uses salt state files to define the state a server needs to be in.

Between the remote execution system, and state management Salt addresses the backbone of cloud and data center management.

Think Puppet on steroids. And done correctly.

At any event, yes, I’m looking for a Debian Developer to sponsor me getting it into Debian proper. If you are an Ubuntu Developer, and could help sponsor me getting it into Ubuntu as well, that would be awesome!

$ dd if=/dev/scd0 of=cdimage.iso # NO!

Or worse yet:

$ cat /dev/scd0 > cdimage.iso # NO!

As you are probably thinking, the problem with the two commands above, is that they provide no error checking while building the image. So, in order to make sure you have all the bits, you need to use another tool, such as using the MD5 hashing algorithm:

$ md5sum /dev/scd0 cdimage.iso
d642d524dd2187834a418710001bbf82  /dev/cdrom
d642d524dd2187834a418710001bbf82  cdimage.iso

Thankfully, the hashes above match. But, what if they didn’t? Then, you get to redo your dd(1) command (or, shudder, cat(1)) from above, and then rerun md5sum(1) to make sure you got all the bits. Doesn’t sound like much fun to me. Thankfully, there is a better way, one which handles the checksum while doing the copy.

You want to use readom(1) (“read optical media”) from the wodim(1) (“write optical disk media”) package. Consider the following command:

$ readom dev=/dev/scd0 f=cdimage.iso # YES!

If readom(1) fails to get the bits during the copy, it will let you know that it’s struggling. If it got all the bits, you know you have them all, because of the error checking during the copy. Sure will save you a lot of time running manual hashes when finished.

Now, what about burning a copy of the ISO image? Surely you use dd(1), yes? Something like:

$ dd if=cdimage.iso of=/dev/scd0 # NO!

NO! Instead, use wodim(1) directly:

$ wodim -v -eject cdimage.iso # YES!

For the same reasons that you want to use readom(1) for creating ISO images from CD/DVD, you want to use wodim(1) for burning ISO images to CD/DVD. What happens if after using dd(1) to create your CD/DVD, the md5sum(1) hash doesn’t line up with the image? You didn’t get all the bits, and created a coaster. Use wodim(1) and should it succeed, you can rest assured that you have all the bits.

So, remember, readom(1) and wodim(1) are the tools you want when creating and/or burning ISO images from the command line. Any other tool, and you’re likely doing it wrong.

The command which(1) (which is often a csh script, although sometimes a compiled binary) is not reliable for this purpose. which(1) may not set a useful exit code, and it may not even write errors to stderr. Therefore, in order to have a prayer of successfully using it, one must parse its output (wherever that output may be written).

Note that which(1)’s output when a command is not found is not consistent across platforms. On HP-UX 10.20, for example, it prints “no qwerty in /path /path /path …”; on OpenBSD 4.1, it prints “qwerty: Command not found.”; on Debian (3.1 through 5.0 at least) and SuSE, it prints nothing at all; on Red Hat 5.2, it prints “which: no qwerty in (/path:/path:…)”; on Red Hat 6.2, it writes the same message, but on standard error instead of standard output; and on Gentoo, it writes something on stderr.

(Quotation and manpage reference additions mine). So, if which(1) is bad news, then what is the “proper” way to determine if a command is in your $PATH? Well POSIX has an answer, and not surprisingly, the command to use is “command”:

The “command” built-in also returns true for shell built-ins. If you absolutely must check only PATH, the only POSIX way is to iterate over it:

There are also Bash built-ins that can be used, should you have Bash installed on your system:

Or:

If you prefer the ZSH (my addition not present in the wiki), as I do, then you can look in the $commands associative array:

I like that at the end of the FAQ, he gives a shell script for using which(1) should it be absolutely necessary. Not only do you have to test for exit code, but you also have to test for common strings in the output, seeing as though which(1) doesn’t always use exit codes properly:

CONCLUSION:
You have many options to find whether or not a command exists in your $PATH, some POSIX, some proper built-ins. Regardless, you should be able to build platform-independent scripts using the proper tools, and using which(1) is not the right tool for the job. Hopefully, this has convinced you of that.

According to the ssh_config(5) manual:

     LocalCommand
             Specifies a command to execute on the local machine after suc‐
             cessfully connecting to the server.  The command string extends
             to the end of the line, and is executed with the user's shell.
             The following escape character substitutions will be performed:
             ‘%d’ (local user's home directory), ‘%h’ (remote host name), ‘%l’
             (local host name), ‘%n’ (host name as provided on the command
             line), ‘%p’ (remote port), ‘%r’ (remote user name) or ‘%u’ (local
             user name).

             The command is run synchronously and does not have access to the
             session of the ssh(1) that spawned it.  It should not be used for
             interactive commands.

             This directive is ignored unless PermitLocalCommand has been
             enabled.

As mentioned, the used of LocalCommand executes a local command after successfully connecting to the server. I figured this would be a great way to print something to the terminal, letting me know whether or not my client just connected to a production machine, a QA machine, or a development machine.

I wanted to use colors, to make it obvious. I don’t want to make the same mistake twice, so I want it painfully clear what machine I just went to. As a result, if I go to a development or home machine, use green. If I enter a QA machine, use yellow. If I enter a production, or other serious machine I probably shouldn’t be on, use red. As a result, I can take advantage of the ANSI escape sequences for color. In case you forgot, here are the colors and modes:

Colors
\e[{attr1;…;{attrN}m

Text attributes
0 Reset
1 Bright
2 Dim
4 Underscore
5 Blink
7 Reverse
8 Hidden

Foreground Colors
30 Black
31 Red
32 Green
33 Yellow
34 Blue
35 Magenta
36 Cyan
37 White

Background Colors
40 Black
41 Red
42 Green
43 Yellow
44 Blue
45 Magenta
46 Cyan
47 White

So, if I were about to SSH to a production machine, I probably want to make it as obvious as possible. Thus, I could print to the terminal, in blinking, bold, red text “PRODUCTION”. I could use the following command:

print "\e[1;5;31PRODUCTIONm\e[0;m"

Notice that at the end of the sequence, I’m resetting the text attributes. This is because if you don’t do this, you will keep the text attributes in your terminal, and that may have an affect on how the text is displayed when in your remote SSH connection.

A possible ~/.ssh/config file could look like this:

Here is a screenshot in action (without the blink):

At any event, here is the configuration for the theme:

Obviously, this won’t look that great on a terminal with a white background (or really any color other than black). And here is the screenshot:

As you can clearly see, the active window you’re under is bold white with red parentheses around the window name. The previous window you were in is marked with a dash ‘-’ (by default). An alert in a terminal will change the text to bold yellow, so long as you’re not in that window (as you can see with the “mutt” window). It ties in nicely with the 88_madcows.theme file for Irssi, and the ZSH theme I built.

I’m new to building Tmux hardstatus lines, so if there is something I should be doing differently, let me know.

If you have thought about attending, you should register. You can see my talk, if you sign up just for the enthusiast package. You can attend a sponsored lunch (and other things) if you upgrade.

Also, I’ll be hosting the PGP keysigning party at the conference as well (a follow-up post is coming). My key is in the “strong set”, so by me signing your key, I’ll also be bringing you into the strong set as well, if you’re currently not in it. It’s all about strengthening the Web of Trust.

At any event, see you at the fest!

These are best practices from the client perspective, not the server. Many of these points you will have already been familiar with, but some of them not as much. If you sit down and think about what you are doing as a client when you SSH to a server, some of the implications mentioned here make sense.

Lastly, security is a big topic, and not something that can be flipped on and off with a switch. Security always starts with the user. Unfortunately, increasing your security could mean making usability more of a pain in the rear. However, as an OpenSSH client, I hope the techniques mentioned here won’t decrease your usability too bad, while greatly increasing your overall OpenSSH security. With that said, let’s get started.

0. Use ECC first, then RSA, then DSA with maximum bit strength.
When generating your OpenSSH keys, you should be aware of the cryptographic algorithms of the OpenSSH server that you are accessing, and what you can and cannot use. As of OpenSSH version 5.7, elliptic curve cryptography is a supported algorithm. It is proving to be a very secure, robust and light crypto algorithm, but it also must be supported by both the client and the remote server. This means that both the client and the server must be relatively new installations of OpenSSH- something that RHEL 6 and earlier, Debian GNU/Linux Stable 6 and earlier, Ubuntu 10.10 and earlier, and likely many other stable releases from other GNU/Linux operating systems, do not support. When you can, use elliptic curve cryptography for your SSH keys.

If ECC is not available on either your client or server, then you should choose RSA as your key’s crypto. DSA suffers from a weakness, where if the host does not have a sufficiently strong pseudorandom number generator (PRNG), then the “random k” could become exposed from the public key, allowing the attacker to build the private key from the public. The PRNG supplied by GNU/Linux (the device file “/dev/urandom”) is sufficiently strong, providing good random data from entropy pools- thus, GNU/Linux generally doesn’t suffer from this problem. Other operating systems could.

A great demonstration of DSA’s weakness can be found in this excellent blog post, where the author demonstrates how trivial it is to build the private key when the “random k value” is known. Fortunately, RSA does not suffer from this weakness, and also allows you to build larger keys than DSA.

If neither ECC nor RSA is supported by the client nor server, as is the case with some archaic proprietary SSH implementations, then DSA is your only option. Unfortunately, it is also the default for OpenSSH when generating a key. So, you should always get into the habit of passing the “-t [type]” switch when generating your keys.

In all 3 cases, where possible, you should choose the maximum bit strength that the algorithm allows. This will put a bit of strain on the client’s CPU, but it will also give you the greatest strength when authenticating on the wire. When generating your key, use the “-b [size]” switch to specify the bit strength.

1. Use SSH keys and make your servers require them.
There are three reasons why you would want to use SSH keys:

1. SSH keys with a passphrase provide two-factor authentication.
2. SSH passwords can be read in plain text on the remote server.
3. SSH keys aren’t subject to brute force dictionary attacks, like passwords.

Let’s cover each in detail. First, two-factor authentication. SSH keys use public key cryptography. You are given a private and public key when generated. During generation, you are asked to provide a passphrase for the private key. DO NOT GIVE IT AN EMPTY PASSPHRASE! By providing a passphrase to your key, you have enabled the two-factor authentication- something you have (the key) and something you know (the passphrase). Using SSH keys can be troublesome, however. As a result, take advantage of the SSH agent for your system to cache the passphrase locally on your client. This will increase your usability by only entering your key passphrase once, and using the agent to login to other servers without providing the key passphrase again.

Second is reading the user password in plain text on the remote SSH server. You may not think this is possible, as everything in OpenSSH is encrypted, right? Wrong. The two Achilles Heel’s in the connection are the client and servers themselves, where the decryption is taking place. Other users, specifically those who have superuser access, can exploit this. To demonstrate this, make a connection to a server that you have superuser access on. Then, from the client initiate a connection that would require your server password, but don’t supply it. Go back to the server, and find the process of the connection, and run an strace on the PID. Go back to the client, and type in your password. Watch, very likely in horror, the password be displayed on your console.

Here is an excellent writeup by Joseph Hall on this very issue. The problem lies in whether or not you trust those who have superuser access on the server. You should only trust them enough to do their job on the server, and nothing more. Just because they have superuser access on the server, doesn’t mean they can be trusted with your banking account information. Because people use the same passwords over and over across multiple accounts, it’s likely that the password sniffed out of the connection is the same password for their email account. Or bank.

Third, as should be obvious, SSH keys aren’t subject to brute force dictionary attacks like passwords are. If you control the SSH server, and require that SSH keys are the mode of authentication, then brute force attacks will be in vain, regardless how long they try. If you have a publicly facing SSH server, you’re likely aware of how hard your server gets hit from attackers all over the world. By forcing key authentication, all those attacks are in vain.

2. Don’t use a blank passphrase on your keys.
This should come as no surprise, but needs to be mentioned. Your SSH keys are something that should be guarded carefully, with sufficient paranoia. If for any reason, your SSH client is compromised, your SSH keys are a way in to the remote servers that you have access to. If your keys are not protected by passphrases, then after scouring your shell history, or SSH config for hosts to connect to, they’re in the SSH server with little effort. You must protect your keys with strong passphrases!

Now, I understand that there are some like me, who use SSH as the connection for nightly local backups, at which point the client will be asked for the passphrase of the SSH key. Because you wish this to be an automated process, and likely when you’re in bed, you won’t be available all the time to provide the credentials. So, you create an SSH keypair that is not passphrase protected. In such a scenario, here is what I would do:

1. Only install this key on the backup SSH server, and no where else.
2. Do not install this key into an account that has superuser access.
3. Do not install this key into the superuser account.
4. On the backup server, change the key entry in the authorized_keys file to only allow connections from that client using that key (documented how below).

The first, second and third points are easy enough to configure, but how do you configure the fourth point, and is it even possible? When you copy the key to the authorized_keys file on the SSH server, it could look something like this:

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzoblHIUARNP5Kq12QwUqxB6T7m8TWti4LIFcvOCa...

Change the beginning of that entry to this:

from="10.19.84.10",command="/home/user/bin/validate.sh" ssh-rsa AAAAB3NzaC1y...

This tells the server that the only connections allowed to use this key can come from “10.19.84.10″, and when the connection is made, a local script is ran called “validate.sh”. Here are the contents of that script for me:

Make sure the script is executable by the user account on the SSH server, and test it before committing to it as part of your backup policy. If the client host is behind a dynamic IP address, or some other variable that prevents it from having a static address, you can remove the “from=” part, but leave the “command=” part. The goal is to minimize damage that can be done with that key.

Again, this is all if you need to perform an automated backup with SSH keys, where a passphrase cannot be performed. In almost every other case, your SSH key should be protected by a good, strong, loaded with entropy passphrase. Using one from a passwordcard is probably best.

3. Use a separate key for every client you SSH from.
I admit that when I first started using SSH, this didn’t make much sense to me. The amount of keys that I generated was a lot, and managing them became somewhat of a pain. Then, when becoming system administrator of a large organization, and controlling upwards of 300 servers, managing that many keys become immensely intense.

I soon came to the realization that it wasn’t that big of a deal. It is trivial to copy the key to the remote host (this can be done with the “ssh-copy-id(1)” command). Further, for work SSH keys, while I am in charge of managing those keys, I should only worry about work keys at work, and home keys at home. In other words, separate the management of the keys.

But the point of this is to NOT copy the same client key to multiple clients. Think about it. By copying the key around to multiple clients, this means that the key is on every server you ever access from those clients, even if some of the other clients cannot access that server directly. Thus, if a client is compromised, and the key is stolen, it’s difficult to know which client was compromised, and all servers that have that key in their authorized_keys file, are subject to compromise. If you have a different key for each client, then those keys are only on the servers that the client has direct access to, and should a client become compromised, you know which servers to saefguard, and damage is minimal.

4. Limit the number of clients you SSH from.
If an attacker can compromise your client, then they can get access to your SSH keys, as they are stored on the filesystem. Further, they may be able to install a keylogger to log passwords and passphrases, including those for your key. Once this information is gained, then all the accounts on the servers the key is installed on are now compromised as well. By limiting the number of clients you SSH from, you minimize the exposure of SSH servers to the attacker.

5. Don’t “chain” or “loop” logins, but do “star”.

The point relates to the one above it, namely limiting the number of hosts you SSH FROM. There are a few scenarios on how you can create SSH sessions:

• Chain: As a client, you make a connection with an SSH server. Then, from that server, you make another connection to a different SSH server. Once, twice, three times or more, you’ve created a chain of connections from the client to the final host.
• Loop: As a client, you make a connection with an SSH server. Then, from that server, you make a connection back to the client you started from, thus creating a loop with your connections.
• Star: As a client, one connection is made to an SSH server. If another connection is needed to a different SSH server, this is made from the client. For each connection, the original client makes it.

The reasons why this is a “best practice” might not be obvious. Let’s start first with chaining connections. If one client is compromised in the chain, the other clients in the chain might possibly be compromised using the same method. Thus, all connections in the chain could become compromised. And a loop is nothing more than a specific instance of a chain.

Of course, this isn’t always possible. Maybe a chain is the only way to get to an SSH server on a private VLAN or behind a DMZ. These connections might be necessary (typically called “jump hosts”), but they should be used with caution, and as little as possible. When you are done with your task, rather than idle the SSH connection, terminate it to minimize exposure, should an attack occur.

When using a jump host, install netcat(1) on the jump host and use the ProxyCommand configuration paramater. Something like this would be sufficient:

Host inaccessiblehost1 inaccessiblehost2
   ProxyCommand ssh accessiblehost nc -q0 %h %p

With star connections, you minimize the risk of compromise with each of your connections, but security is maximized, even if extra work is required on your end. For example, if you wish to transfer data from one SSH server to another, this may mean bringing the data to the client, then sending it to the destination server.

6. Consider disabling the SSH server on the client you are connecting from.
This rule shouldn’t apply only to the SSH server, but to any daemon you have running on your machine. If you don’t need a service running, then turn it off. By doing so, you minimize the attack vector and greatly increase your security. OpenBSD prides itself in having only two remote holes in the default install in a heck of a long time, but then it also doesn’t have any services running on a default install either.

So, for SSH, because you followed rule #5, and you aren’t doing “chain” or “loop” connections from your client, then this means that you’re not connecting to your own machine that you started your initial connection from. I understand that this isn’t always possible. I have 300+ SSH servers at work, and I can’t possible turn off the SSH server on a host, just because I’m connecting from it. But then, do I need an SSH server on my virtual laptop? Not likely, and if there are things I need, I can turn it on, do my work, then turn it off.

7. Don’t ignore SSH host key warnings.
When you install an SSH server for the first time, it creates a server public and private keypair. This set of keys is what is used for the encryption and decryption between your client and the SSH server. Thus, all traffic that you are sending to that server, is done because you trust that the SSH server key presented to you, is indeed the key of the server itself. So, what happens when you connect a different time, and your client warns you that the public SSH server key has changed? Should you trust the connection?

Depends. Usually, it’s due to the fact that you either reinstalled the SSH server or reinstalled the operating system. In this case, you can move forward, so long as you know that the key presented is the new key from the server. However, someone could setup a proxy SSH server, for the intent of gaining system passwords. Your client will warn you the key is different, and you should pay attention to the warning.

So, in your SSH client config, you generally should not set the following:

Host *
    StrictHostKeyChecking no

Default is “yes”, and it should stay yes in your config. OpenSSH is verbose enough for your benefit. Try to take advantage of it, rather than silence it, or ignore it. After all, it’s your data.

8. Be wary of using X11 forwarding.
While taking advantage of the ability to run remote X applications locally on your client is nice, it has some very grave drawbacks. For example, it’s common for people to launch a remote browser, such as Firefox. What isn’t realized, is that you’re providing usernames and passwords through the browser, which could be caught on the remote machine from which the app is running (taking advantage of X events, for example).

There is a time and a place for X11 forwarding, and I have benefited by it, but generally, it’s not needed. Set the following in your SSH client config, and only enable it when needed:

Host *
    ForwardX11 no

9. Don’t use agent forwarding.
As with X11 forwarding, you are giving ultimate trust to the remote host when forwarding your local agent. If that connection makes a connection to an account with superuser privileges, then a user on the remote SSH server could hijack your agent by connecting to the socket the SSH server creates.

Also, on the client initiating the connection, the superuser must be trusted, which isn’t always the case. The local superuser can access the agent socket on the client, and attack. Thus, it’s generally not a good idea to enable agent forwarding. Thus, as it is by default, the following should be sent in your client SSH config:

Host *
    ForwardAgent no

At any event, in graphical user interfaces, you’ll see floppy disks that represent saving a file to disk. It’s very, very common, yet I doubt many of the younger generation has ever used one. So, how are they supposed to know what it does?

So, here’s a poll for you to take. You’ll need to take the poll at my site, as it uses Ajax calls to the page, which means it won’t work in your RSS reader, or on any other site. I’m just curious to see what falls out.

1. The command line interface.
2. Various shells, including their script syntax.
3. Builtin programming language support for many languages.
4. Common Unix utilities, such as grep, rsync, ssh, lsof, and others.
5. All the supported filesystems (ZFS, Ext4, Reiser, UFS, etc.).
6. Overall rock-solid stability and reliability.
7. Lack of viruses, trojans, and other malware.
8. Tremendous networking capabilities (PPoE, TCP/IP, etc.).
9. Bulletproof firewall software.
10. Overall builtin security in general (MACs, PAM, etc.).
11. Quality user/group management.
12. System resource usage.
13. Both vertical and horizontal scaling.
14. Portability.
15. Plain text configuration files.
16. Open source kernel and user-space software.
17. Based on standards (POSIX, FHS, LSB, etc.).
18. Vast selection of software choices (various text editors, MUAs, etc.)
19. Simplicity in software design- do one thing, and do it well.
20. Mind-blowing hardware support.
21. Support for hundreds of languages and locales out of the box.

shred -n 3 -v /dev/sda

It works well enough. However, it doesn’t display a real useful progress meter, other than how far it’s done in the wipe, thus leaving it up to you to figure out the speed, while filling up your back scroll in the process. There must be a better way.

I used to “leave my mark” (much like a dog marks a fire hydrant), however, this is quite slow. There are other methods, such as using /dev/urandom, but the entropy from urandom relies on SHA1. While fast, it’s not the speed demon that is AES or other algorithms. There’s /dev/zero, but how do I get random bits from zeros? And more importantly, does it push the drive to it’s bandwidth threshold? Of course, I’ve heard about DBAN, but I’ve had issues with it booting on certain hardware. Lastly, I would like to have a good progress meter as the data goes down on the drive.

Here’s a solution that a friend of mine in an IRC channel suggested:

openssl enc -aes128 -k "foo" < /dev/zero | pv -trb > /dev/sda

The great thing with this command is two fold:

1. It’s fast. It pushes the drive to as fast as it can write data.
2. It provides a convenient progress meter with “pv”

Again, I’m shredding drives with pseudorandom data. I’m not too concerned about the security of the bits going down on the platter. Per corporate regulation, I need to do 3 passes, and I’m confident that the bits coming out of the pipe from OpenSSL using AES-128 will be sufficient. So, for doing 3 passes, I can script it easily enough:

for I in 1 2 3; do
    openssl enc -aes128 -k "$I" < /dev/zero | pv -trb > /dev/sda
done

That works. 1 drive down, 24 to go…

If you have various ways you shred your drive, let me know, and I’ll post it below.

First, I like the command line. I like minimalist function and design. It’s always pulled me in. So, this solution is a command line solution. With that said, the command line doesn’t work well for viewing images, does it? This can be problematic for RSS feeds. I understand this. However, since using Newsbeuter for the time I have, I’ve found that I read a lot of blogs that are 100% images, and when push comes to shove, it’s adding a lot of noise to my RSS signal. As a result, I’ve removed a great deal of those feeds, and haven’t missed them. Again, going back to minimalism, I read fewer feeds now, and the signal is much, much higher. However, the link to the RSS feed is in the post itself, so you can follow the link from Newsbeuter to the browser of your choice if you wish. This will be the minority of the time for myself.

Second, this solution is not only using Newsbeuter, but also Mutt. Really, any mail client will work, but I like Mutt, so I’ll be using it in the solution here. Further, we’ll be tying Reader and Buzz together to make a few things work. But, everyone hates Buzz, right? Well, I don’t. I love the tight integration it has with Gmail and with Reader. Yeah, the design is lacking, that’s for sure. And some of the keystrokes are completely bass-ackwards compared to reader, and can cause problems with Gmail. However, because of the integration, and the reply-by-mail feature, which we’ll take advantage of here, you’ll find you won’t miss your Google Reader much (unless you absolutely have to view images in your feeds).

With that said, let’s begin. First, you need to install Newsbeuter 2.2 or greater and an mail client (Mutt, as already mentioned, will be used here). After installing Newsbeuter, you need to configure it to sync with your Google Reader account. Here’s what I set in my ~/.newsbeuter/config:

# setup google reader sync support
urls-source "googlereader"
googlereader-login "username@gmail.com"
googlereader-password "my-password"
googlereader-flag-share "S"
googlereader-flag-star "s"

Make sure you read the documentation about the Google Reader support. Now, fire up Newsbeuter. After authenticating, you will notice that it has populated your feeds, and you can start pulling down the updates to unread items. Here’s a screenshot of my Newsbeuter “home”:

You now have access to your starred items, your shared items, popular items and people that you follow (what others are sharing), but not the comments. That’s okay, we’ll cover how to get access to that in a second. However, if you have access to your starred items and shared items, how do you share or star an item in Newsbeuter? This is done through flags. Newsbeuter has a flagging system that allows a custom categorization of feeds by the user. The flag must be a letter, ether uppercase or lowercase, A-Z and a-z. You set the flag by pressing “Ctrl-e”, then entering the flag you wish to set. Now, you’ll notice that in my config, I have two options that are for flags: “googlereader-flag-share” and “googlereader-flag-star”. If I press “S” for my flag, then it will share the item. If I press “s” for my flag, then it will star the item. Test it, then log into the Google Reader interface to see it work.

Now, when I would share items, they usually were shared with a comment to encourage discussion with those who are following me (or anyone who can read the shared item). However, in Newsbeuter, there is no way to comment on a flagged item directly. Further, when you read items that your friends share, and they have comments with them, how do you add your own commentary? Well, I have a solution, but it doesn’t involve Newsbeuter. Rather, it requires that you use Buzz and an email client.

With Buzz, you can add your Google Reader shared items as a connected page. This means that whenever you share an item on Reader, it will show up as a new item in Buzz. Further, Buzz will send you an email of all the items you’ve posted to your wall, as well as your items that have been commented on. When you receive the email, the body of the message might be something like this (text copied and pasted from Mutt):

Date: Tue, 31 Aug 2010 13:59:49 -0700 (PDT)
From: Aaron Toponce <z12dhj4psprvstctz23vcxg5wleozbfi104@gmail.com>
To: Aaron Toponce <aaron.toponce@gmail.com>
Subject: Buzz from Aaron Toponce

[-- Autoview using /usr/bin/elinks -force-html -dump ''/tmp/mutt.html'' --]
   [1]Aaron Toponce – Google Reader   Aug 31, 2010
   [2]Man Already Knows Everything He Needs To Know About Muslims - [3]Aaron
   Toponce's Friends' Facebook Links
   [4]Man Already Knows Everything He Needs To Know About Muslims
   Source: [5]www.theonion.com
   [6][IMG]
   SALINA, KS—Local man Scott Gentries told reporters Wednesday that his
   deliberately limited grasp of Islamic history and culture was still more
   than sufficient to shape his views of the entire Muslim world.
       I seem to recall that in earlier times it was only the court jester
   who could get away with speaking unwelcome truth to power. Perhaps that is
   why comedy often forces us to see things about ourselves we don't want to
   see.
   –––––
   Reply to this email to add a comment to this post.
   Link to this post:

http://www.google.com/buzz/115784859563110525602/BaqgGreVqdT/Man-Already-Knows-Everything-He-Needs-To-Know

References

   Visible links
   1. http://www.google.com/profiles/aaron.toponce
   2. http://www.facebook.com/profile.php?id=1068226280&v=wall&story_fbid=124010764315178
   3. http://www.facebook.com/posted.php
   4. http://www.theonion.com/articles/man-already-knows-everything-he-needs-to-know-abou,17990/
   5. http://www.theonion.com/
   6. http://www.theonion.com/articles/man-already-knows-everything-he-needs-to-know-abou,17990/

Notice the “From” address. It’s a long unique string that gives you access to comment on the post directly. As mentioned in the email, all you have to do is reply to the email, adding your commentary, and it will be posted as a comment on the shared item. You’re actually commenting on a Buzz post, not a Reader post, however, because you tied Reader and Buzz together, they become one and the same. Also, Buzz is smart enough to strip signatures and attachments from the reply, so your email signature and S/MIME PGP attachment will also be stripped. Just make sure you trim your email appropriately, so you’re not adding anything irrelevant to your comment (trim everything out of your reply).

Go back to Reader, pull up the shared item, and notice your comment on the post. Of course, you can comment on others shared items by sharing it first (the “S” flag for me), then replying to the mail sent from Buzz. When Buzz gives the ability to be notified of everything that your friends are posting, then you shouldn’t have to share it also. However, I usually reshare what others have shared, so this isn’t a show-stopper for me, even if it is less than perfect.

UPDATE: I forgot to mention Newsbeuter’s tagging feature. In Google Reader, I have separated my feeds into folders. However, when I sync with Reader, it seems all my feeds are in one massive “river of news”. I don’t like this, and wish the folders were preserved. Well, with Newsbeuter, it is through tags. When in the main window, press “t” and it will take you to your “tags”, or folders as Reader would call it. At which point, you can read only what specific topic you’re interested in. However, I do know that Google Reader supports tagging beyond their folders. I don’t know of a way to apply Reader tags inside Newsbeuter.

So, when installing on a desktop or laptop, I’m slightly annoyed by the default package sets that are installed. I understand why they’re chosen, and I’m definitely not arguing with the decisions made, however, some of the packages just aren’t for me. So, on my wiki, I’ve been documenting what should be installed and removed on every Debian GNU/Linux installation I make. As soon as the install finishes, I’ll login to the system, pull up my wiki, and copy and paste the following commands as root (trying to keep the horizontal scrolling in your browser to a minimum):

# aptitude install abiword apt-file checkconfig chromium-browser clusterssh deborphan flashplugin-nonfree git-core \
gmrun gnumeric htop network-manager openbox openssh-server python-docutils rst2pdf screen tango-icon-theme \
vim vim-gnome xfce4 xfce4-icon-theme xfce4-terminal xscreensaver-data-extra zsh zsh-doc

# aptitude purge dasher gnome-accessibility gnome-accessibility-themes gnome-mag gnome-orca gok nano \
openoffice.org{,-base,-base-core,-calc,-common,-core,-draw,-emailmerge,-evolution,-filter-binfilter,-filter-mobiledev} \
openoffice.org{-gcj,-gnome,-gtk,-help-en-us,-impress,-java-common,-math,-officebean,-style-galaxy,-style-tango,-thesaurus-en-us,-writer} \
xserver-xorg-input-{all,synaptics,wacom} xserver-xorg-video-{all,apm,ark,ati,chips,cirrus,fbdev,i128,i740,intel,mach64,mga,neomagic} \
xserver-xorg-video-{noveau,nv,openchrome,r128,radeon,radeonhd,rendition,s3,s3virge,savage,siliconmotion,sis,sisusb,tdfx,trident,tseng,v4l,voodoo}

First, I will identify which xserver-xorg-video driver I’ll need for the hardware I’m installing to, and remove that from the purge, even though the VESA driver always works just fine for me, as I’m certainly no gamer, and it’s never let me down in my desire to power any video card at any resolution and frequency I’ve encountered. I’ll also leave the Synaptics driver in place when installing to a laptop.

Second, I realize that this isn’t for everybody. Most people won’t care about having OpenOffice.org or the GNOME accessibility packages installed. I don’t need them, they free up hard drive space for me, and when running updates, downloads are much faster. So, this works for me.

Lastly, I have a hard time deciding between using GNOME and Openbox for my default desktop. I like the GDM for logging in, and I like the power management features that GNOME brings to the table. However, I like the minimalist approach to Openbox, and its configuration capabilities. I login to both from time-to-time, just in case you noticed that I’m installing Openbox, but also removing GNOME packages that would only come from a default GNOME install.

I’ve gotten tired of executing this on every install, and wondered if there was something more I could do. So, I mentioned my frustration on Identi.ca, not really looking for a reply, but one came through at any event (thread here). The reply was to use DPKG to get a list of the software that is installed when I’m finished with my install and purge, then use that list during the next install.

So, how to do this? Well, after you have your system installed the way you want, with all the packages installed and purged to suit your needs, run the following command:

$ dpkg --get-selections \* > packages.txt

You now have a text file with all the packages that are installed on your system. So, when doing a fresh Debian install at a later date, it’s trivial to get these packages installed, so I don’t have to do the install and purge copy/paste that I was doing before. After installing only the base, and nothing more, login to the system, copy the packages.txt file you created via SCP or some other method to the filesystem, and run the following commands (as root):

# apt-get update
# dpkg --set-selections < packages.txt
# apt-get -u dselect-upgrade

You now have your system installed with exactly the packages you want installed, and nothing more. Not only that, but the latest version as well. No need to update after you login (Windows, you could learn something here).

To me, this is beautiful. This is a simplistic way to clone a Debian system, without using utilities like Norton Ghost or CloneZilla, either though both have their place in administration. I just love the simplicity and elegance of this. To me, this makes administration fun. When I can solve simple problems with core system tools, I'm a happy admin.

Long live Debian.

Recently, we had some virtual servers take a dive, because the disk array they were on took a dive, and with it, all the storage that these VMs were relying on for their OS and other needs. So, we’ve been building these machines back up from backup on a new NetApp SAN. Everything has been working well with Bare Metal Restore for Windows VMs, but we were struggling getting it to work with RHEL. So, I got the idea to boot up a Fedora 12 Live CD, install the NetBackup client in the live environment, and do a restore from there, then reboot into the newly built machine.

Works like a charm! So, I documented the steps necessary to make it work. Realize, that this was documented for my place of employment. Of course, I’ve completely changed the hostnames, IP addresses, and other details that would be specific to work. However, if you deploy this in your environment, then you’ll need to make the necessary architectural changes that fit your needs. Also, this should work with Fedora 13, the latest release at the time of this writing, but Fedora 12 works well for us, and because it’s used just for this purpose, we see no pressing need to use the latest. Also, Debian or Ubuntu, or some other GNU/Linux based operating system might work as well, but NetBackup doesn’t support these operating system necessarily, and it is expecting the filesystem layout that is common with Red Hat based operating systems. Regardless, Fedora 12 is being used as a tool here, not because of any loyalty or fanboyism.

So, here’s the tutorial:

Fedora can be used as a temporary live environment for NetBackup when doing restores to RHEL machines. Just boot from the Fedora ISO or CD, and make sure you have configured networking appropriately, and that you can get out to the Internet. After that, follow the instructions below. This does assume that you have a base, bare-bones, RHEL install that matches the partitioning or volume layout of the previous install that you’re restoring. This must be completed first, because at the end of the tutorial, you will be mounting these devices, and restoring only the files, to these mount points.

First, the liveuser can get root access without any password. However, for this tutorial, you will need to set a password for root, so go ahead and do that:

[liveuser@localhost ~]$ su -
[root@localhost ~]# passwd
Changing password for user root.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
[root@localhost ~]# 

At this point, we need to setup the hostname and networking correctly. We have set aside the ‘netbackup.example.com’ domain name for the Linux restore system in our DNS environment, and the IP address that has been set aside is ’10.19.84.254′. In the terminal as root, set the hostname:

[root@localhost ~]# hostname netbackup.example.com
[root@localhost ~]# export HOSTNAME="netbackup.example.com"

Now get networking configured with NetworkManager. In the status bar of the desktop, there is an icon that looks like two computers. Right click this, and click “Edit Connections…”. Click “Auto eth0”, click the “Edit…” button, click the “IPv4 Settings” tab, and set the parameters as necessary:

Method: Manual
Address: 10.19.84.254
Netmask: 255.255.255.0
Gateway: 10.19.1.254
DNS servers: 10.19.2.192, 10.19.3.192, 10.19.4.192
Search domains: example.com

Click “Apply…”, and enter the password for root, then close the “Network Connections” widget. The icon looking like two computers should have a red “X” next to it now. This means it’s not online. Left-click the icon, and click “Auto eth0” to get it back online. Back in your terminal, you should be able to verify that the networking has been set correctly:

[root@localhost ~]# ip a show eth0
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:1a:4a:06:14:24 brd ff:ff:ff:ff:ff:ff
    inet 10.19.84.254/24 brd 10.5.31.255 scope global eth0
    inet6 fe80::21a:4aff:fe06:1424/64 scope link
        valid_lft forever preferred_lft forever
[root@localhost ~]# ping -c 4 mediaserver.example.com
PING mediaserver.example.com (10.19.84.60) 56(84) bytes of data
64 bytes from mediaserver.example.com: icmp_seq=1 ttl=64 time=0.942 ms
64 bytes from mediaserver.example.com: icmp_seq=2 ttl=64 time=0.230 ms
64 bytes from mediaserver.example.com: icmp_seq=3 ttl=64 time=0.193 ms
64 bytes from mediaserver.example.com: icmp_seq=4 ttl=64 time=0.212 ms

--- mediaserver.example.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3002ms
rrt min/avg/max/mdev = 0.193/0.394/0.942/0.316 ms

In order for the NetBackup client to communicate with the media server (mediaserver.example.com – 10.19.84.60), we need to make a modification to our /etc/hosts file, and add it’s IP address and hostname:

[root@localhost ~]# echo "10.19.84.60      mediaserver.example.com mediaserver" >> /etc/hosts

With networking correctly set, we need to start up the SSH server:

[root@localhost ~]# service sshd start
Generating SSH1 RSA host key:                              [  OK  ]
Generating SSH2 RSA host key:                              [  OK  ]
Generating SSH2 DSA host key:                              [  OK  ]
Starting sshd:                                             [  OK  ]

Because this is a temporary live environment, we’re not interested in SELinux nor any firewall. So, let’s get those disabled first:

[root@localhost ~]# getenforce
Enforcing
[root@localhost ~]# setenforce 0
[root@localhost ~]# getenforce
Permissive
[root@localhost ~]# service iptables stop
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Unloading modules                                [  OK  ]

At this point, you should verify that you can SSH to the Fedora live environment from another host. It should work. However, we still have more work to do. Before we can get the NetBackup client installed, we need to install a few packages. So, in your terminal, type the following:

[root@localhost ~]# yum install glibc-2.11-2.i686 libacl-2.2.47-5.fc12.i686 libstdc++-4.4.2-7.fc12.i686

This will download about 30MB of packages and metadata necessary for installation. Depending on your Internet connection, this might take a while. If you don’t want to wait, you could change the repo files in /etc/yum.repos.d/, and use a faster mirror. Or, you can wait. After the packages are installed, and everything configured, we should be able to install the NetBackup client. To do this, we need to SSH to mediaserver.example.com, and send the client over. You can do this from the terminal you’ve already been working in:

[root@localhost ~]# ssh mediaserver
The authenticity of host 'mediaserver (10.19.84.60)' can't be established.
RSA key fingerprint is 14:d3:50:f1:27:d5:14:ab:c2:ca:51:fa:7e:5a:98:4c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'mediaserver,10.19.84.60' (RSA) to the list of known hosts.
root@mediaserver's password:
[root@mediaserver ~]# 

Now transfer over the client. First, mediaserver.example.com will likely have the wrong SSH key for this box on file, as you might have done this more than once, and every time you boot into the Fedora 12 Live CD, and start up the SSH server, a different key will be generated. You will need to delete it:

[root@mediaserver ~]# cd /usr/openv/netbackup/client/Linux/RedHat2.6/
[root@mediaserver RedHat2.6]# ./sftp_to_client netbackup.example.com root
Connecting to 10.19.84.18...
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
89:db:1f:12:f9:8f:76:38:63:6e:54:75:7a:43:ed:9e.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending key in /root/.ssh/known_hosts:4
RSA host key for netbackup.example.com has changed and you have requested strict checking.
Host key verification failed.
Couldn't read packet: Connection reset by peer

sftp connection to netbackup.example.com failed.

Notice, that it says the offending key is on line 4 of /root/.ssh/known_hosts. So, I’ll delete line 4 from that file:

[root@mediaserver RedHat2.6]# sed -i 4d /root/.ssh/known_hosts

Now, try again:

[root@mediaserver RedHat2.6]# ./sftp_to_client netbackup.example.com root
Connecting to netbackup.example.com...
The authenticity of host 'netbackup.example.com (10.19.84.254)' can't be established.
RSA key fingerprint is 4d:74:e6:66:ed:d9:bc:dc:3a:71:41:51:18:58:da:e9.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.19.84.254' (RSA) to the list of known hosts.
root@netbackup.example.com's password: 

sftp completed successfully.

The root user on netbackup.example.com must now execute the command
"sh /tmp/bp.22802/client_config [-L]".  The optional argument, "-L",
is used to avoid modification of the client's current bp.conf file.
[root@mediaserver RedHat2.6]# exit
[root@localhost ~]#

Now that we are back in the live environment, we can execute the command as given us from mediaserver.example.com. Of course, the name of the file transferred over might be different. First, let’s get xinetd running:

[root@localhost ~]# service xinetd start
Starting xinetd:                                           [  OK  ]
[root@localhost ~]# sh /tmp/bp.22802/client_config

It will spit out a bunch of files and directories that it’s installing for the client, and will reload the xinetd server. All that is left, is to start the NetBackup client, and we’re finished!

[root@localhost ~]# service nbclient start
NetBackup SAN Client Fibre Transport daemon started.

Verify that the default NetBackup ports 13722, 13724, 13782 and 13783 are bound to the system:

[root@localhost ~]# lsof -Pan -i tcp | grep 137
xinetd   3261 root    5u  IPv6  22914      0t0  TCP *:13782 (LISTEN)
xinetd   3261 root    6u  IPv6  22915      0t0  TCP *:13722 (LISTEN)
xinetd   3261 root    8u  IPv6  22916      0t0  TCP *:13724 (LISTEN)
xinetd   3261 root    9u  IPv6  22917      0t0  TCP *:13783 (LISTEN)

At this point, you should be able to login to the NetBackup Administrator Console, find the netbackup.example.com client, and verify that they can talk. The only thing that you should be aware of, is that you need to mount all logical volumes and partitions to their respective mount points BEFORE the restore! This is important! When in the NetBackup Administrator Console, and doing the restore, you restore the files to a specific mount point. In our case, it’s going to be ‘/’. This means, that if the host has separate /var, /u01, /, /home and other mount points, they all need to be mounted to their respective mount points BEFORE doing the restore!

[root@localhost ~]# mount /dev/work/root /mnt
[root@localhost ~]# mount /dev/work/var /mnt/var
[root@localhost ~]# mount /dev/work/u01 /mnt/u01
etc

CONGRATULATIONS! At this point, you’re ready to start restoring the data using NetBackup in the Fedora 12 live environment. Just remember, that when you’re in the NetBackup Admin Console, you need to restore the data to netbackup.example.com, and the directory path you need to use should be /mnt, not /, obviously.

I hope someone on the remote corners of the Internet finds this tutorial helpful. If so, let me know if the comments. If there are any spelling errors or I’ve missed something giving away detail of my works, also let me know so I can make the corrections. Thanks!