% find /path -exec rm -rf {} \;

While certainly functional, it’s not optimal. If there are thousands of files (as is often the case at my job), this command is slow, slow, slow. The reason being are all the excessive fork() and exec() calls for each pass with rm(1). Instead, you could optimize find(1) by using “-delete”:

% find /path -delete

This is much more optimal, but it has one VERY nasty side effect. If you place “-delete” in the wrong spot in your find(1) command, you could delete all the files listed under “/path” before processing the necessary logic. From the find(1) manual:

Warnings: Don’t forget that the find command line is evaluated as an expression, so putting -delete first will make find try to delete everything below the starting points you specified. When testing a find line that you later intend to use with -delete, you should explicitly specify -depth in order to avoid later surprises. Because -delete implies -depth, you cannot usefully use -prune and -delete together.

One nice benefit of “-delete”, however, is the proper handling of NUL characters in your filename, such as spaces, tabs or the newline character. Thankfully, there is another option, which is not only supported in GNU/Linux, but also in FreeBSD (and perhaps others):

% find /path -print0 | xargs -0 rm -rf

This avoids the excessive fork() and exec() system calls from our first command, and doesn’t have the nasty side effects of “-delete”. Further, because of “-print0″ as a find(1) argument, and “-0″ with xargs(1), we can handle files properly with NUL characters. Time the three commands above, and you’ll see that the last is most optimal.

We can squeeze some extra juice out of the command, though. All we need to do is cd(1) to the directory we wish to operate our find(1) command on:

% cd /path && find . -print0 | xargs -0 rm -rf

Working with removing millions of files (yes, I do actually remove that many, often), I have found this latest find(1) command to be the most optimized in terms of sheer speed. It moves. You may find the same results as I.

FYI.

I placed a hidden message in my email headers for a bit (I’ve since stopped, for various reasons). I considered it an “Easter Egg” of sorts, waiting for someone to notice. Here is what I placed in the headers:

Crypto-Challenge: iVBORw0KGgoAAAANSUhEUgAAADwAAAA8AQMAAAAAMksxAAAABlBMVEX
       ///8AAABVwtN+AAAAtklEQVQokXXQMQ6CMBQG4McCiykXMPEKsuEiV2nCBdoLWNgN
       Xqld7MYZeoQSFgbisyZWER//9E1//vcAYhQOvkB0X3AQotBAoZ5lV9hpA63ZxCP5Q
       SiE5N28QpjRxT0rhFzi7BWUlx3LQvMH+x1jx7IEAoer8hVqR4Crhp1fhf9QI976tF
       oAIGlYWTkCCr3I7yTCpTkaDQTqWUhjtFtCNizNgEbbZxMFDnIYrHUEwjPRnywQiHk
       CI/3gDHrryF4AAAAASUVORK5CYII=
Crypto-Hint: image/png

Quickly, you should identify the “Crypto-Challenge” header as base 64-encoded string. The hint says it’s an image, of type PNG. So, the following Python code should do the trick:

Running that code with the base 64-encoded string above gives the following image:

Scanning the QR code reveals the text “42″, of which most geeks should recognize as “The Answer to the Ultimate Question of Life, the Universe, and Everything”.

Of course, steganography isn’t encryption. It’s security by obscurity, which isn’t security, where a message is hidden by obscuring it through some means. Wikipedia has a great article on it at https://en.wikipedia.org/wiki/Steganography.

What can you do with hidden messages in images (or vice versa, as in the case with my email “Easter Egg”)? Well, for one, you can easily get around email attachment restrictions. For example, take a ZIP archive. Perhaps some organization blocks email with .zip attachments. Why not convert the archive to base 64, then convert the result to an image. You might end up with something like this:

Your final image might end up like:

In our case, I just created a file from /dev/urandom, zipped it up, and converted to an image. Thus, the reason the data in the image appears so random. More structured files will show actual structure in the final image. Also, notice the string of black at the bottom as a result of our padded zeros to adjust for a square image, without losing data.

Of course, to get back to the archive, you just need to reverse the process of converting the image to a base64 string, then back to the original file. Now, I’m no Python expert, and I realize there is much more to add to the code, such as “try/except” blocks for testing files, writable directories, etc. The point of the code was just to demonstrate an overall algorithm.

Hopefully, this is of some interest to some of my readers. I’m open to code improvements. Thanks to https://diablohorn.wordpress.com/2010/12/04/whats-in-a-picture for use of the code above.

Case in point: I just filled out a form for a survey to “enter to win a $1000 shopping spree). You know, the crap that you constantly get bombarded with at the checkout stand when the cashier gives you your receipt. I always ignore them, but then thought to myself “I’ll never win if I don’t at least try”, so I gave my first survey a go. At the end of the survey, it asked for my email address. I figure they’ll sell it for marketing purposes, and I have a Google Mail address, so I’m not really that worried about the SPAM (their SPAM filters are amazing). But, I would like to track who they are selling my address to. So, I gave them the following address:

aaron.toponce+survey-provider@gmail.com

To which, I received an error that the email address is not a valid address. AHEM! Yes it is, and it’s this lack of support for standards that I’m talking about. My email address was rejected, yet it’s perfectly legal according to RFC 5322. You see, according to that RFC, I get the following flexibilities with my email address:

• ASCII upper and lower case letters (a-z & A-Z).
• ASCII digits 0-9
• ASCII characters !#$%&’*+-/=?^_`{|}~
• ASCII dot (.) so long as the local part of the address does not contain the dot consecutively, and it does not start with a dot.
• ASCII characters ” ” (space) and “(),:;<>@[\] are allowed with certain restrictions.

So, I could have the following email addresses, all of which are perfectly legit according to the RFC:

• “[Aaron Toponce]“@gmail.com
• a&t@gmail.com
• aaron.toponce+business@gmail.com
• aaron’s-travel-agency@example.travel
• {atoponce}@gmail.com

Yet, these will get ejected outright in most web forms I’ve come across. Specifically interesting is the .travel TLD. I’ve had web forms enforce TLDs that are less than 4 characters, which is absolutely absurd for the .travel and .museum TLDs. I’m guessing one of two things is happening with these web forms:

1. The developer used the regular expression [A-Za-z0-9_\-\.]+@[A-Za-z0-9\-\.]+ for validating addresses.
2. There is absolute denial for the use of “plus-addressing” as a DEA.

I’m guessing the first is more likely the scenario than the second. Regardless, Of course, when we’re talking about the rules of RFC 5322, we’re no longer talking about regular expression syntax. We’re talking about grammar. If your page is designed in PHP, Python, CGI, or whatever, you should use a real parser for parsing the email address, rather than reinventing the wheel yourself. What’s unfortunate, is this disease of not properly parsing valid email addresses is found in some big companies and sites too, not just the little guys.

Now, Google COULD provide true DEAs, such as Yahoo! Plus does with their subscribers, However, I should be able to create an DEA with an already existing email address, rather than creating completely new ones, because people refuse to conform to the standards. So Google, if you’re reading (I know you are), you may want to consider proper DEAs, seeing as though “plus addressing” isn’t working, and it is important to some.

Stéphane Bortzmeyer has already blogged about this, and he uses the #ral hashtag on Identica and Twitter to vent his frustrations, which stands for “Refus d’Adresses Légales” or “Rejection of Legal Address”. Well, I’ve determined that I will be doing the same, although I’ll bacronym the hashtag to “Rejected And Legal”, along with the url to the site that refuses to adhere to the RFC.

So, the question is: do you trust the short URL? Well, I’ve generally gotten into the habit of asking people to expand the shortened url for me if on IRC, email or IM, and it’s worked just fine. But, I got curious if there was a way to do it automagically, and thankfully, you can use wget(1) for this very purpose. Here’s a “quick and dirty” approach to expanding shortened URLs (emphasis mine):

$ wget --max-redirect=0 -O - http://t.co/LDWqmtDM
--2011-10-18 07:59:53--  http://t.co/LDWqmtDM
Resolving t.co (t.co)... 199.59.148.12
Connecting to t.co (t.co)|199.59.148.12|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://is.gd/jAdSZ3 [following]
0 redirections exceeded.

So, in this case “http://t.co/LDWqmtDM” is pointing to “http://is.gd/jAdSZ3″, another shortened URL (thank you Twitter for shortening what is already short (other services are doing this too, and it’s annoying- I’m looking at you StatusNet)). So, let’s increase our “–max-redirect” (again, emphasis mine):

$ wget --max-redirect=1 -O - http://t.co/LDWqmtDM
--2011-10-18 08:02:12--  http://t.co/LDWqmtDM
Resolving t.co (t.co)... 199.59.148.12
Connecting to t.co (t.co)|199.59.148.12|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://is.gd/jAdSZ3 [following]
--2011-10-18 08:02:13--  http://is.gd/jAdSZ3
Resolving is.gd (is.gd)... 89.200.143.50
Connecting to is.gd (is.gd)|89.200.143.50|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://wiki.ubuntu.com/UbuntuOpenWeek [following]
1 redirections exceeded.

So, in this case, the link finally points to https://wiki.ubuntu.com/UbuntuOpenWeek. I’m familiar enough with the Ubuntu Wiki, that I know I should be safe visiting the initial shortened URL. If you want to add this to a script or shell function, then you can get a bit more fancy:

$ expandurl() { wget -O - --max-redirect=$2 $1 2>&1 | grep ^Location; }
$ expandurl http://t.co/LDWqmtDM 1
Location: http://is.gd/jAdSZ3 [following]
Location: https://wiki.ubuntu.com/UbuntuOpenWeek [following]

In this case, our “expandurl()” function takes two arguments: the first being the URL you wish to expand, and the second being the max redirects. You’ll notice further that I added “-0 -” to print to STDERR. This is just in case you give too many redirects, it will print the content of the page’s HTML to the terminal, rather than saving to a file. Because you’re grepping for “^Location”, and sending the HTML to your terminal anyway, technically you could get rid of the “–max-redirects” altogether. But, keeping it in play does seriously increase the time it takes to get the locations. Whatever works for you.

UPDATE (Oct 18, 2011): After some comments have come in on the post, and some discussion on IRC, there is a better way to handle this. According to the wget(1) manpage, “-S” or “–server-response” will print the headers and responses printed by the FTP/HTTP servers. So, here’s the updated function that you might find to be less chatty, and faster to execute as well:

$ expandurl() { wget -S $1 2>&1 | grep ^Location; }
$ expandurl http://t.co/LDWqmtDM
Location: http://is.gd/jAdSZ3 [following]
Location: https://wiki.ubuntu.com/UbuntuOpenWeek [following]

Perfect.

$ dd if=/dev/scd0 of=cdimage.iso # NO!

Or worse yet:

$ cat /dev/scd0 > cdimage.iso # NO!

As you are probably thinking, the problem with the two commands above, is that they provide no error checking while building the image. So, in order to make sure you have all the bits, you need to use another tool, such as using the MD5 hashing algorithm:

$ md5sum /dev/scd0 cdimage.iso
d642d524dd2187834a418710001bbf82  /dev/cdrom
d642d524dd2187834a418710001bbf82  cdimage.iso

Thankfully, the hashes above match. But, what if they didn’t? Then, you get to redo your dd(1) command (or, shudder, cat(1)) from above, and then rerun md5sum(1) to make sure you got all the bits. Doesn’t sound like much fun to me. Thankfully, there is a better way, one which handles the checksum while doing the copy.

You want to use readom(1) (“read optical media”) from the wodim(1) (“write optical disk media”) package. Consider the following command:

$ readom dev=/dev/scd0 f=cdimage.iso # YES!

If readom(1) fails to get the bits during the copy, it will let you know that it’s struggling. If it got all the bits, you know you have them all, because of the error checking during the copy. Sure will save you a lot of time running manual hashes when finished.

Now, what about burning a copy of the ISO image? Surely you use dd(1), yes? Something like:

$ dd if=cdimage.iso of=/dev/scd0 # NO!

NO! Instead, use wodim(1) directly:

$ wodim -v -eject cdimage.iso # YES!

For the same reasons that you want to use readom(1) for creating ISO images from CD/DVD, you want to use wodim(1) for burning ISO images to CD/DVD. What happens if after using dd(1) to create your CD/DVD, the md5sum(1) hash doesn’t line up with the image? You didn’t get all the bits, and created a coaster. Use wodim(1) and should it succeed, you can rest assured that you have all the bits.

So, remember, readom(1) and wodim(1) are the tools you want when creating and/or burning ISO images from the command line. Any other tool, and you’re likely doing it wrong.

The command which(1) (which is often a csh script, although sometimes a compiled binary) is not reliable for this purpose. which(1) may not set a useful exit code, and it may not even write errors to stderr. Therefore, in order to have a prayer of successfully using it, one must parse its output (wherever that output may be written).

Note that which(1)’s output when a command is not found is not consistent across platforms. On HP-UX 10.20, for example, it prints “no qwerty in /path /path /path …”; on OpenBSD 4.1, it prints “qwerty: Command not found.”; on Debian (3.1 through 5.0 at least) and SuSE, it prints nothing at all; on Red Hat 5.2, it prints “which: no qwerty in (/path:/path:…)”; on Red Hat 6.2, it writes the same message, but on standard error instead of standard output; and on Gentoo, it writes something on stderr.

(Quotation and manpage reference additions mine). So, if which(1) is bad news, then what is the “proper” way to determine if a command is in your $PATH? Well POSIX has an answer, and not surprisingly, the command to use is “command”:

The “command” built-in also returns true for shell built-ins. If you absolutely must check only PATH, the only POSIX way is to iterate over it:

There are also Bash built-ins that can be used, should you have Bash installed on your system:

Or:

If you prefer the ZSH (my addition not present in the wiki), as I do, then you can look in the $commands associative array:

I like that at the end of the FAQ, he gives a shell script for using which(1) should it be absolutely necessary. Not only do you have to test for exit code, but you also have to test for common strings in the output, seeing as though which(1) doesn’t always use exit codes properly:

CONCLUSION:
You have many options to find whether or not a command exists in your $PATH, some POSIX, some proper built-ins. Regardless, you should be able to build platform-independent scripts using the proper tools, and using which(1) is not the right tool for the job. Hopefully, this has convinced you of that.

In my “~/.muttrc”, I have my accounts separated into two files, so I can set specific options for each account that don’t affect the other. As a result, here is the relevant parts of my ~/.muttrc:

# ~/.muttrc
folder-hook "personal.mail.tld" "source ~/.mutt/personal.rc"
folder-hook "work.mail.tld" "source ~/.mutt/work.rc"
source ~/.mutt/personal.rc # use this as the default account when Mutt executes

So, I have two accounts: “personal.mail.tld” and “work.mail.tld” (those are actual URLs to your accounts, whether they be GMail, Yahoo!, or whatever. Change as necessary). So, let’s take a loot at the relevant parts of “~/.mutt/personal.rc” and “~/.mutt/work.rc”:

# ~/.mutt/personal.rc
source ~/.mutt/gpg.rc

# ~/.mutt/work.rc
source ~/.mutt/smime.rc

Obviously, I’m sourcing external files. The personal account sources the gpg.rc config, while the work account sources the smime.rc config. Both of these configs can be found as example files in “/etc/Muttrc.d/” on Debian. From there, I made my edits.

Obviously, for both configs, I will have needed to setup cryptographic keys. For GnuPG, I will need to generate and save off my public key pair. There is a wealth of documentation on the internet that discusses this, so I won’t cover that here. I’ll just assume you have it created already.

For S/MIME, you will need to generate an OpenSSL certificate, signed by a centralized certificate authority. For myself, I chose Comodo. It was easy and quick. I had my SSL cert in less than 5 minutes, and it’s good for a full year, and it was free.

Once you get the email about the certificate, when you click the link to open it in your browser, it will install your certificate in the browser. No big deal. Just navigate to the certificate using your browser’s menu, and backup the certificate (both the private and public keys) to some local directory on your machine. It should have “.p12″ as its extension. Then, copy the certificate to the machine that will be running Mutt.

On the machine that you will be running Mutt from, you will need to issue a few commands to get your environment setup correctly before you can start using the certificate. You will need OpenSSL installed before you can begin:

$ smime_keys init

This should create “~/.smime/”, “~/.smime/certificates/” and “~/.smime/keys/”. After which, you will need to copy “ca-bundle.crt” to “~/.smime/”. “ca-bundle.crt” is provided by a number of packages, and you can find it on the internet should it not already be installed. After that is copied, type the following:

$ smime_keys add_p12 /path/to/your-backed-up-cert.p12

You will be asked for the passphrase for the private key in the .p12 file, as well as a new passphrase for encrypting the key in your database. It doesn’t matter if you use the same passphrase both times. This passphrase is not recoverable, so make sure you remember it. You will also be asked what you want to call the certificate; it makes no difference to anyone what you call it, but you can’t have two certificates with the same nickname.

Make note of the hash of your key. You can retrieve this hash from the line “added private key” followed by a path and a hexadecimal number ending in “.0″. Find the line in your .muttrc that says:

# ~/.mutt/smime.rc
set smime_default_key="12345678.0"

and change “12345678.0″ to the hash for your key that you took note of. The only thing left to do is to install the Comodo root certificates. These are already installed on your system in “/etc/ssl/certs/Comodo*” and “/etc/ssl/certs/COMODO*”. For each of those root certs (I’m actually not sure which is needed for your personal cert), you need to issue the following command:

$ smime_keys add_root /etc/ssl/certs/Comodo_Secure_Services_root.pem

You have now successfully setup your environment to use S/MIME with Mutt. Now let’s look at the configs. For the gpg.rc config, I kept everything default except for the following:

# ~/.mutt/gpg.rc
set pgp_good_sign="^gpg: Good signature from"
set crypt_autosign="yes"
set crypt_replysign="yes"
set crypt_replysignencrypted="yes"
set pgp_auto_decode="yes"
unset smime_is_default

For the smime.rc config, again I kept everything default except for the following:

# ~/.mutt/smime.rc
set smime_is_default="yes"
set crypt_autosmime="yes"
set pgp_autosign="no"
set smime_timeout="300"
set crypt_autosign="yes"
set crypt_replyencrypt="yes"
set crypt_replysign="yes"
set crypt_replysignencrypted="yes"
set crypt_verify_sig="yes"
set smime_default_key="12345678.0" # you should have already changed this

You are now ready to roll. When you change to your personal account, it should use PGP/MIME by default, and when you change to your work account, it should use S/MIME by default.

According to the ssh_config(5) manual:

     LocalCommand
             Specifies a command to execute on the local machine after suc‐
             cessfully connecting to the server.  The command string extends
             to the end of the line, and is executed with the user's shell.
             The following escape character substitutions will be performed:
             ‘%d’ (local user's home directory), ‘%h’ (remote host name), ‘%l’
             (local host name), ‘%n’ (host name as provided on the command
             line), ‘%p’ (remote port), ‘%r’ (remote user name) or ‘%u’ (local
             user name).

             The command is run synchronously and does not have access to the
             session of the ssh(1) that spawned it.  It should not be used for
             interactive commands.

             This directive is ignored unless PermitLocalCommand has been
             enabled.

As mentioned, the used of LocalCommand executes a local command after successfully connecting to the server. I figured this would be a great way to print something to the terminal, letting me know whether or not my client just connected to a production machine, a QA machine, or a development machine.

I wanted to use colors, to make it obvious. I don’t want to make the same mistake twice, so I want it painfully clear what machine I just went to. As a result, if I go to a development or home machine, use green. If I enter a QA machine, use yellow. If I enter a production, or other serious machine I probably shouldn’t be on, use red. As a result, I can take advantage of the ANSI escape sequences for color. In case you forgot, here are the colors and modes:

Colors
\e[{attr1;…;{attrN}m

Text attributes
0 Reset
1 Bright
2 Dim
4 Underscore
5 Blink
7 Reverse
8 Hidden

Foreground Colors
30 Black
31 Red
32 Green
33 Yellow
34 Blue
35 Magenta
36 Cyan
37 White

Background Colors
40 Black
41 Red
42 Green
43 Yellow
44 Blue
45 Magenta
46 Cyan
47 White

So, if I were about to SSH to a production machine, I probably want to make it as obvious as possible. Thus, I could print to the terminal, in blinking, bold, red text “PRODUCTION”. I could use the following command:

print "\e[1;5;31PRODUCTIONm\e[0;m"

Notice that at the end of the sequence, I’m resetting the text attributes. This is because if you don’t do this, you will keep the text attributes in your terminal, and that may have an affect on how the text is displayed when in your remote SSH connection.

A possible ~/.ssh/config file could look like this:

Here is a screenshot in action (without the blink):

At any event, here is the configuration for the theme:

Obviously, this won’t look that great on a terminal with a white background (or really any color other than black). And here is the screenshot:

As you can clearly see, the active window you’re under is bold white with red parentheses around the window name. The previous window you were in is marked with a dash ‘-’ (by default). An alert in a terminal will change the text to bold yellow, so long as you’re not in that window (as you can see with the “mutt” window). It ties in nicely with the 88_madcows.theme file for Irssi, and the ZSH theme I built.

I’m new to building Tmux hardstatus lines, so if there is something I should be doing differently, let me know.

Announcing Penny Red, a Python solution to integrate Hashcash into Mutt, licensed under the GPLv3. I chose the name “Penny Red” for a two reasons:

1. Penny Red was the second British postal stamp which was debuted in 1841, the first being the Penny Black stamp. The color was changed, so the black cancellation mark could be easily seen on a red stamp, versus the previous black stamp. Because the goal of the project is to implement a payment system through minted tokens, and a previous solution had existed in Perl (without documentation, mind you), this seemed appropriate.
2. Unfortunately, in the English language, the words “red” and “read” have the same sound, yet different spelling. After reading your email, you would say “I have read my mail”. Because it had postage attached in the headers, you read mail that was paid for. So, “Penny Red” is a play on “Penny Read”

The great thing with the Python scripts in Penny Red is their portability. The “mint_hashcash.py” script reads a file as a passed argument, and then writes to the headers. The “verify_hashcash.py” script reads the mail from STDIN and prints the message and the verified tokens back to STDOUT. Nothing specific about Mutt is in either script! As a result, as long as the MUA supports STDIN and STDOUT with each message, and modifying the headers, these scripts can be used. I guess what I’m saying is, I would like to extend these scripts to Pine, Alpine, Gnus (although not really necessary as Gnus supports Hashcash out of the box), Elm, and others, without forking the project. Penny Red should be able to support multiple MUAs.

At any event, I just wanted to get this post out there, seeing as though I just barely setup the project. I’m pretty excited. Should be fun, and give me something to do after graduation.

First, the necessary changes to your “~/.muttrc” config. The script relies on the “X-Hashcash:” header (as well as the “Hashcash:” header- I guess there was some discrepancy or something about X-headers being deprecated, or something) not being weeded out. It must be displayed if present. This way, the script can actually see the token in the header, and process the logic of checking if it’s valid. If you hid the hashcash headers, then the script won’t execute, and as a result, won’t add anything to the token database. We use the $display_filter variable to execute the Python script, and show the results to the pager. Here’s the changes you will need to make:

# file: ~/.muttrc
ignore *                                    # draconian header weed - recommended
unignore from date subject to cc user-agent # standard headers unignored - recommended
unignore x-hashcash hashcash                # required

You will also need to set the $display_filter variable. I like having my theme consistent, so I’ve also added color to my theme to show the Hashcash verification at the top of the mail:

# file: ~/.muttrc
set display_filter="/path/to/verify_hashcash.py"    # required
color body brightyellow default "^\\[--.*Hashcash*" # recommended

Now with that set, all we need to do is install the Python script, and we’re ready to go. When looking over the code, you’ll notice that it’s creating a database of spent tokens. I’ve placed the database in ~/.mutt/, seeing as though I only have Mutt working with Hashcash at the moment (and it’s really the only MUA I use these days). If you have something else that uses Hashcash tokens, in an already existing database, you may want to make the necessary modifications to the Python script, so it’s pointing to the right file. Also, we’re only interested in keeping track of tokens minted for us personally, not all tokens we can find in the headers. Lastly, this Python script is only working for one email address. If you have multiple emails, as I do, you’ll have to either use a primary email to verify the tokens against, or modify the Python script to support checking tokens under multiple accounts.

With that said, here’s the script:

One thing I do find odd about the code above, is the hashcash binary, when checking tokens, prints to STDERR rather than STDOUT. I’m guessing this could change in the future, so that’s something that will need to be watched out for.

So far, the Python script has been working flawless for me. I haven’t noticed a single hiccup, and it’s fast, which it should be. However, standard disclaimers apply, such as not coming with any warranty, blah, blah, blah, and if you find any bugs, or have any enhancements (such as supporting multiple email addresses), I’m all ears. At any rate, I hope you find it helpful, should you wish to get into Hashcash with Mutt.

I wanted to used Hashcash with Mutt, for nothing more than a curiosity to see if it generates any discussion, and to see if people notice. Further, I’m a big crypto advocate, and while Hashcash isn’t exactly crypto, it’s highly related to it, and uses it. Regardless, I wanted to see if I could seamlessly tie in Hashcash with Mutt, both for minting and verification.

I make the following assumptions:

• You have Hashcash installed.
• You have Mutt installed.
• You have Python 2.5 or later installed.

With that, let’s continue.

What is Hashcash?

Hashcash is a proof-of-work protocol, with the motivation of eliminating SPAM. The idea is simple, although we’ll cover it in greater detail. The TL;DR version, is using Hashcash is about generating tokens, and attaching them to the header of the message you’re sending. The recipient checks the token for verification. If it passes, then you must not be SPAM, as you’ve put your computer through enough strenuous work to prove it. If the token fails, you could be SPAM, or you just did it wrong. Either way, it’s no guarantee that you’re not a spammer.

Now, the detailed version.

The premise behind Hashcash is that some certain mathematical results are difficult to discover but easy to verify. Factoring large numbers is one such example. So Hashcash is some sort of challenge for your CPU. For example, I say that you cannot comment on my blog, unless you can find the two prime factors of some large number. Once you’ve paid the work through CPU cycles, and provide the numbers, I multiply them together to see if it gives the result. The work was difficult for you to find, but simple for me to verify. Hashcash is based on this principle.

Using a lot of CPU cycles to answer a question non-interactively is one way to beat spammers at their own game, which Hashcash hopes to address. Rather than looking for large prime factors though, Hashcash is looking for a SHA1 sum of a unique string where the first set of characters in the sum are zero. Because SHA1 can provide 2^160 possible hashes before a collision is found, it shouldn’t take too much work to find such a string. In other words, the search space is large enough for the work to be reasonable. However, the search space is also large enough, that given a certain criteria, it can be difficult to find the requested string.

Let’s look at an example. Consider the SHA1 string:

00000528ba02c4da777839db269fde97de56ddf7

The first 20 bits of the string are zero. So, this is one such SHA1 hash that would work. The questions are, how did we find it, and what string did we use to generate it? Well, because the first 20 bits of the string are zero, this means that I should only have to search up to 2^20 possible hashes to find the string I’m looking for. On a moderate system within the last 10 years or so, this should take under a second. Indeed, the SHA1 hash string above was done on an AMD Athlon(tm) XP 1800+ with only 384 MB of PC133 RAM, and it was calculated in 940 milliseconds. The string, or in this case, our “Hashcash token” that generated that SHA1 hash is:

1:20:110321:aaron.toponce@gmail.com::Ci2v7/gsBbYY/dhk:0BTS

You can verify this easily enough:

echo -n 1:20:110321:aaron.toponce@gmail.com::Ci2v7/gsBbYY/dhk:0BTS | sha1sum
00000528ba02c4da777839db269fde97de56ddf7  -

So, if that is indeed a Hashcash token, then what are all the fields, and how are they used when searching for a SHA1 hash that starts with zeros? Well, the breakdown is thus:

1. The version number.
2. The claimed bit value.
3. The date (and time) a stamp was minted.
4. The resource for which a stamp is minted.
5. Extensions that a specialized application may want.
6. A random salt.
7. The suffix.

Hashcash has undergone two revisions thus far. The version number shows the claimed version of the protocol being used. With the latest release of Hashcash, this is “1″. “0″ has shown to have some limitations.

The claimed bit value it telling the verifier of the token how many zeros the resulting SHA1 hash should have. The default is 20-bits, but this can be changed. If the bit value is too low, then it becomes trivial for spammers to reproduce with minimal effort. If the bit value is too high, it may take some considerable time for your CPU to mint the token.

The timestamp is the date when the token was minted. We can take advantage of this for expiry. When the verifier downloads the token, he can keep it in a database. This is useful to hopefully prevent someone else from claiming the same token. However, holding on to tokens indefinitely could be cumbersome, so we can set expiration dates on when the tokens expire, to help manage the database. The default expiration is 28 days.

The resource for which the token is minted is generally the email address of the recipients. However, it’s flexible enough to be anything. It could be a URL to a webpage, some string of text, or anything that uniquely identifies a certain resource. We’ll be using email addresses as our resource.

The fifth field is generally left blank, but the protocol specifies that this is used for extensions that any specialized application may want.

The salt is used just like it is used anywhere else. Here, we wish to make our token unique for our resource. The email address and timestamp might not be unique enough to prevent two minted tokens from being the same. So, we add a random, hopefully unique salt here to the token. Once the salt is chosen, just like the timestamp and the resource, it won’t change for that token. The real grunt work is done with the next field. But, the goal with the salt is to just eliminate the possibility that two people send me an email on the same day, and the minted token is the same. If each salt is randomly chosen, then the tokens will differ.

As stated, this last field is where the real grunt work of your CPU lies. Every previous field is statically set upon instantiation. It is only this field that changes until we find the SHA1 sum that produces the correct number of zeros that we’ve asked for. Fortunately, the suffix is sequential, starting at zero, and working it’s way up, base 64, until the appropriate suffix attached to this string gives us the SHA1 we want. As mentioned, if our bit size is 20 bits, by default, then we will only have to work our way through 2^20 possible suffixes, on average, to find the right string that gives us the SHA1 with 20 bits of leading zeros.

The security of the token comes from the fact that there should only be one collision for every 2^160 possible strings in SHA1. Let’s not get tied up in the cryptanalysis. Suffice it to say, that SHA1 is still holding well in cryptographic circles, and for all practical purposes, we are interested in the amount of work it takes for the CPU to find the right token. When SHA1 becomes broken, and cryptanalysis shows it’s no longer secure enough for today’s applications, Hashcash is flexible enough to move to a different cryptographic hashing algorithm by just changing the protocol version. In the meantime, SHA1 works.

Also, given today’s multicore CPUs, and Moore’s Law, 20 bits might not be enough work for the CPU to crunch through. Remember, the idea is to beat spammers at their own game. If they can produce mass valid tokens for each spam mail they send, then we should make sure that our bit strength is stronger. Right now, 20 bits seems to be strong enough, but maybe moving it to 24 or 28 bits in the future might be necessary. Just remember the amount of time it takes for your CPU to calculate the work needed for the token.

How Hashcash works with email

When the token is minted, we wish to add the token to our mail header. We can add a token for each recipient in the “To:” and “Cc:” lines (with special care on how the headers are modified with Bcc: recipients (generally solved by sending separate emails with individual tokens)). What is added to the mail header is the following:

X-Hashcash: 1:20:110321:aaron.toponce@gmail.com::Ci2v7/gsBbYY/dhk:0BTS

For those using Hashcash, including antispam utilities such as SpamAssassin, looking for the X-Hashcash header in the mail, then verifying that the minted token is valid takes a fraction of a second. But, as we discussed earlier, minting the token takes some time. 20 bits for me takes just under a second. So, now imagine a SPAM ring with control of thousands of zombie computers infected with trojans. There are only 86400 seconds in a day, so if it takes one second to mint a valid token, then a single computer could only send out at most 86400 mails in a day, a far cry from the millions it wishes to send out. So, while it doesn’t completely eliminate the zombie nets from sending mass emails, it does slow them down considerably.

Yet, for those of us who only send a couple dozen mails per day, it’s trivial for our CPU to do the work, and doesn’t slow us down any. Further, those receiving our messages can verify the token is valid in a fraction of the time it took to mint it. As the receiver, you set the price of the token you wish to validate as antispam. Again, 20 bits seems sufficient enough for today, but 24 or 28 bits might need to be your standard in the future.

For those of you who are not using Hashcash, you can simply ignore the header, and move on with your life. It won’t slow your mail experience down any, and shouldn’t get in the way of reading or replying to it.

Mutt Configuration

Now that we have a decent understanding of how the Hashcash protocol works, let’s see how we set this up with Mutt. Thankfully, this is rather straight forward. All you need to add to your ~/.muttrc is:

set edit_headers="yes"                 # required
set editor="/path/to/mint_hashcash.py" # required
set askcc="yes"                        # recommended, but not required

This will allow you to edit the mail headers when composing your message with the editor of your choice. The editor is chosen for you in the Python script. I use Vim, so it’s hardcoded to that. Feel free to change it to your preferred editor of choice. I should also mention at this point that I have only found a way to automate minting tokens with Mutt. I haven’t found a way yet, although I’m working on it, to automate the process of verifying tokens, and storing tokens in a database, with Mutt. Hopefully, a followup post can come to fruition when I figure it out (after all, what’s the point of minting tokens if you can’t verify others’ tokens yourself?).

The script is here:

The code is fairly straight forward. Parse the email headers using RFC 822, grab the necessary email addresses, and mint the tokens as necessary. You’ll quickly note that “Bcc:” addresses aren’t supported, as minting tokens for those addresses would give them away in the header. I could send separate emails for each Bcc recipient, putting their own token in the header, but I’m not motivated enough to write that code, and I rarely, if ever, use blind carbon copy.

I’ve been running the above code now for 2-3 days, trying to break it as best I can, and I haven’t come across anything. If you do notice a bug, or something sloppy in the code, let me know, and I’ll make a best attempt at addressing it. However, it seems to be working quite well. The only thing I would mention, is if you have a lot of emails in the “To:” or “Cc:” fields, it may take some time to mint all those tokens. Just let it do its thing. It will get you back to Mutt. I promise.

First, my ZSH prompt was already developed from the outset with that in mind. So, no additional hacking has been needed on that. There are some elements that I’m not too terribly excited about. I don’t care for the dark blue directories on a black background, and I don’t care for the yellow character devices on a white background. Using the Tango scheme for gnome-terminal makes both of those scenarios much more tolerable. However, I do have additional items that I want to put in my prompt, but that will be for a later post. Also, my GNU Screen hardstatus line also needed to be compatible. This wasn’t that big of a deal, as I only needed to apply some colors to a few elements. Hhere’s a couple screenshots showing both black and white backgrounds, and how the ZSH prompt inside looks in each. Note the GNU Screen hardstatus line is at the bottom of the terminal.

From Desktop Screenshots

From Desktop Screenshots

Second, my Irssi theme also needed to work well with both. As I’ve already blogged before, I was really, really impressed with the madcows theme. However, I didn’t care for a few elements, so I started hacking it, making my own changes. I’ve tried keeping the true nature of the theme, while still adding my own style. The theme was already largely compatible with my ZSH prompt colors, it just needed some adjustments here and there, before I was totally satisfied. Further, it looked like crap using a white background, so this needed some hacking as well. I think I’m overall happy with the result, although I’m sure I’ve missed many things (like DCC, or CTCP), so there’s likely much more hacking to go before it’s perfect. However, for the general day-to-day chat, it’s 95% there. Screenshots for both backgrounds below:

From Desktop Screenshots

From Desktop Screenshots

In a nutshell, the themes are compatible with xterm-color support on most terminals. Mainly, I’m using bold and normal weights on red, green, yellow, black, white and blue. Anything else takes the default color of the terminal itself, whether it be the foreground text or the background. So, as long as your TERM variable is set to “xterm-color” or something better, you should be okay.

This post wouldn’t be complete without the source for you to try it out. Here’s a compressed tarball for giving it a shot, and reporting anything you find in the comments, if you like.

Cheers!

You now have the following variables available in the shell script namepace: RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN, BLACK, WHITE, BOLD_RED, BOLD_GREEN, BOLD_YELLOW, BOLD_BLUE, BOLD_MAGENTA, BOLD_CYAN, BOLD_BLACK, BOLD_WHITE, RESET. Using these variables, you can manipulate the output from “echo” and “printf” for your script. For example, here’s a screenshot using “echo” to print red, green and blue text to the screen. Notice that I’m using the “RESET” variable after the blue text to reset my prompt text back to normal. This may or may not be necessary, depending on how you configured your prompt, but it’s not a bad habit to get into.

Thought this might be helpful to the larger scripting community or for those sysadmins, such as myself, who would like a little variety added to their script output.

At the Utah Open Source Conference, I gave a BOF on Unix shells. The turnout was good, and we had a great discussion. I presented on my default prompt for ZSH, showing all the hidden features of the prompt. However, I had forgotten that I had removed battery status from my prompt, because I was depending on APM, which is no longer compiled in the kernel. A couple people have asked me since then why I’m depending on APM and not ACPI. I don’t have an answer, other than that was just what I coded. So, last night, I put up an ACPI implementation, and it works great. As with the APM implementation, if the battery percentage is less than 15%, the percentage display is red. If it’s less than 50% but greater than 14%, it’s yellow, and if it’s less than 100% but greater than 49%, it’s blue. If it’s 100%, or the tool “acpi” is not installed, then it doesn’t show up. Here’s a screenshot below:

While hanging out in our local LUG channel for the Ogden Area Linux Users Group, I got talking with Seth about prompts. He decided to change his, including adding the dog from Nethack randomly “moving” in the prompt. He also mentioned changing the color of the path if the present working directory was not writable. I really liked this idea, and decided to implement it in my prompt. Here’s a screenshot of that in action:

I change the path color to yellow if the present working directory is not writable, as it’s noticeable enough to catch your attention, but subtle enough to not get in the way, and be distracting.

As usual, if you want the source, here it is. Yes, it’s public domain, as mentioned in the code, so have at it.