Image of the glider from the Game of Life by John Conway
Skip to content

Boycott GoDaddy

I generally don’t jump on boycotting bandwagons, usually because they are severely misguided and misinformed, and they’re usually interested in spreading FUD more than just reporting the issue at hand. However, on December 29th, 2011, I will be transferring all of my 15 domains away from GoDaddy, because they support the SOPA and Protect IP bills. You can read more about this at http://www.reddit.com/r/politics/comments/nmnie/godaddy_supports_sopa_im_transferring_51_domains/. Further, there is a boycott site for boycotting GoDaddy, where you can pledge that you will be moving your domains. This site is found at http://godaddyboycott.org/.

December 29th is the day, if I don’t feel the itch to do it before then.

2011 12 23 General
Personal Comments (9) Shortlink

Expand URLs in Irssi

If you’re an IRC junkie, and spend hours a day in Irssi, then this post might be useful to you.

It’s all the rage these days to shorten URLs with fancy URL shortening services. Heck, even I have one. They are certainly nice to have, when links are exceptionally long, such as search result URLs, and just the mere wrapping from one line to the next breaks the URL (not to mention, any additional characters added in the line break, such as spaces, other characters, etc.). I’ve used, and still use, link shortening services for IM, IRC, email, Identi.ca, Twitter, etc., only when I suspect the link could break as a result of line wrapping. I use them sparingly, and only use them if they provide a preview feature, giving the link to the preview.

While they have their advantages, they certainly come with a cost. Link rot is a very real concern, should the link shortening service go offline. You can nest shortened links in each other, concealing JavaScript/CSS mouse hovers. They can contain all sorts of nasties, and you don’t know what you’re getting into, unless you use some sort of software to expand the URL for you, before you actually follow the link. I’ve already blogged about using a simple shell function to expand shortened URLs (post at http://pthree.org/2011/10/18/use-wget1-to-expand-shortened-urls/). Well, now it’s time for Irssi to automatically provide the function for me.

Presenting https://github.com/jcande/Expand-URLs. This is a simple Irssi script that will identify URLs in a given notice, whether in private or in public, and expand them using the http://longurl.org service (I think a patch for doing the lookup without a 3rd party should probably be submitted, as any 3rd party expanding service might go offline).

For me, this script is exceptionally valuable, because I connect to a local Bitlbee instance with Irssi, and use Bitlbee to connect to Twitter. Unfortunately, Twitter wants to track your clicks with their http://t.co service. Every link longer than 19 characters (20 for HTTPS) submitted to Twitter is automatically shortened with this wrapper. They claim that the service is to identify malicious links, and prevent them from being posted, should one be identified. But certainly, a company the size of Twitter can do so much more with this new “service”. They could track what links are clicked and when. They can use this information to identify what stuff you’re interested in, and when you use the service. They can track who clicks the link by IP or ISP. Of course, it would be foolish to not sell this information to advertisers, to target additional advertising on Twitter or other sites, based on this info.

At any event, this is one of the few Irssi scripts that I find really, really useful for day-to-day. It makes the Twitter timeline a bit chatty, now that lengthy URLs are being shown, and a few break due to line wrapping. And that is a pain, no doubt. But, the vast majority of links don’t break, and it’s nice seeing where I’ll be taken when visiting the link. Keeping Twitter from tracking me, despite the occasional link breakage, is worth it.

P.S.: There is also a WeeChat script at http://www.weechat.org/files/scripts/expand_url.pl.

2011 12 22 irssi Comments (0) Shortlink

A Note About Removing Files With find(1)

I’ve seen on the internet, and elsewhere, that when there are too many arguments for rm(1) to handle, that the following command will suffice:

% find /path -exec rm -rf {} \;

While certainly functional, it’s not optimal. If there are thousands of files (as is often the case at my job), this command is slow, slow, slow. The reason being are all the excessive fork() and exec() calls for each pass with rm(1). Instead, you could optimize find(1) by using “-delete”:

% find /path -delete

This is much more optimal, but it has one VERY nasty side effect. If you place “-delete” in the wrong spot in your find(1) command, you could delete all the files listed under “/path” before processing the necessary logic. From the find(1) manual:

Warnings: Don’t forget that the find command line is evaluated as an expression, so putting -delete first will make find try to delete everything below the starting points you specified. When testing a find line that you later intend to use with -delete, you should explicitly specify -depth in order to avoid later surprises. Because -delete implies -depth, you cannot usefully use -prune and -delete together.

One nice benefit of “-delete”, however, is the proper handling of NUL characters in your filename, such as spaces, tabs or the newline character. Thankfully, there is another option, which is not only supported in GNU/Linux, but also in FreeBSD (and perhaps others):

% find /path -print0 | xargs -0 rm -rf

This avoids the excessive fork() and exec() system calls from our first command, and doesn’t have the nasty side effects of “-delete”. Further, because of “-print0″ as a find(1) argument, and “-0″ with xargs(1), we can handle files properly with NUL characters. Time the three commands above, and you’ll see that the last is most optimal.

We can squeeze some extra juice out of the command, though. All we need to do is cd(1) to the directory we wish to operate our find(1) command on:

% cd /path && find . -print0 | xargs -0 rm -rf

Working with removing millions of files (yes, I do actually remove that many, often), I have found this latest find(1) command to be the most optimized in terms of sheer speed. It moves. You may find the same results as I.

FYI.

2011 12 20 Linux
Scripting Comments (6) Shortlink

Steganography

I have been familiar with steganography for a number of years. In fact, back when I was in middle school, I developed a fascination for encryption, and hiding messages, mostly so I could pass notes back and forth to classmates during class. It wasn’t long before I found “invisible ink”, which is a form of steganography. While I’m certainly no expert on the subject, I decided to have a bit of fun with my email.

I placed a hidden message in my email headers for a bit (I’ve since stopped, for various reasons). I considered it an “Easter Egg” of sorts, waiting for someone to notice. Here is what I placed in the headers:

Crypto-Challenge: iVBORw0KGgoAAAANSUhEUgAAADwAAAA8AQMAAAAAMksxAAAABlBMVEX
       ///8AAABVwtN+AAAAtklEQVQokXXQMQ6CMBQG4McCiykXMPEKsuEiV2nCBdoLWNgN
       Xqld7MYZeoQSFgbisyZWER//9E1//vcAYhQOvkB0X3AQotBAoZ5lV9hpA63ZxCP5Q
       SiE5N28QpjRxT0rhFzi7BWUlx3LQvMH+x1jx7IEAoer8hVqR4Crhp1fhf9QI976tF
       oAIGlYWTkCCr3I7yTCpTkaDQTqWUhjtFtCNizNgEbbZxMFDnIYrHUEwjPRnywQiHk
       CI/3gDHrryF4AAAAASUVORK5CYII=
Crypto-Hint: image/png

Quickly, you should identify the “Crypto-Challenge” header as base 64-encoded string. The hint says it’s an image, of type PNG. So, the following Python code should do the trick:

1
2
3
4
# assuming the 'img_string' variable is the actual base64 string above
f = open('crypto-image.png','w')
f.write(img_string.decode('base64'))
f.close()

Running that code with the base 64-encoded string above gives the following image:

Scanning the QR code reveals the text “42″, of which most geeks should recognize as “The Answer to the Ultimate Question of Life, the Universe, and Everything”.

Of course, steganography isn’t encryption. It’s security by obscurity, which isn’t security, where a message is hidden by obscuring it through some means. Wikipedia has a great article on it at https://en.wikipedia.org/wiki/Steganography.

What can you do with hidden messages in images (or vice versa, as in the case with my email “Easter Egg”)? Well, for one, you can easily get around email attachment restrictions. For example, take a ZIP archive. Perhaps some organization blocks email with .zip attachments. Why not convert the archive to base 64, then convert the result to an image. You might end up with something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
from PIL import Image
import base64
import math

# function to return max image size
def get_size(size):
    width = height = int(math.ceil(math.sqrt(size/3)))
    diff = int(((width * height) * 3) - size)
    return (width, height, diff)

# open our binary non-image file
f = open('archive.zip','rb')

# convert the binary to a base64-encoded string
enc_bytes = base64.b64encode(f.read())
f.close()

# get file size to hold data (square)
(w,h,d) = get_size(len(enc_bytes))

# pad with zeros, if necessary
if d > 0:
    for i in range(d):
        enc_bytes += ('\0')

# create our final image
img = Image.frombuffer('RGB',(w,h),enc_bytes,'raw','RGB',0,1)
img.save('image.png')

Your final image might end up like:

In our case, I just created a file from /dev/urandom, zipped it up, and converted to an image. Thus, the reason the data in the image appears so random. More structured files will show actual structure in the final image. Also, notice the string of black at the bottom as a result of our padded zeros to adjust for a square image, without losing data.

Of course, to get back to the archive, you just need to reverse the process of converting the image to a base64 string, then back to the original file. Now, I’m no Python expert, and I realize there is much more to add to the code, such as “try/except” blocks for testing files, writable directories, etc. The point of the code was just to demonstrate an overall algorithm.

Hopefully, this is of some interest to some of my readers. I’m open to code improvements. Thanks to https://diablohorn.wordpress.com/2010/12/04/whats-in-a-picture for use of the code above.

2011 12 15 Python
Scripting
Security Comments (2) Shortlink

Burgers As A Service

There is this burger shop near my home that makes the most amazing burgers, fries and shakes. Bar none. The burgers, of which there is quite the variety, each have their own “secret sauce” that gives each burger its own unique flavor. The fries also have various dipping sauces you can order, each of which are “secret sauces”. Lastly, the shakes, which seems to have a never ending array of flavors, each have their own “secret recipe” to the flavor. Because of these trade secrets, the burgers, fries and shakes are outstanding!

It’s more than just taste too. Portions are epic. They have the “Big Ben” burger, which cut in half would produce two Big Macs from McDonald’s. Then there’s the “Double Ben”, with two patties and the “Triple Ben” with three patties. Add on the amount of fries, and the size of the shake, and you could easily feed a family of four with one order of the Triple Ben.

Lastly, the service is amazing. Every time I’ve visited, I’ve gotten outstanding service from the employees, and the turn around time on preparing my meal is fast. Maybe not as fast as a “fast food” joint, but certainly not as long as your standard dine-in restaurant either. As a result, I recommend Burger Bar in Roy, Utah to anyone and everyone. If you’re a burger, fries and milkshake lover like I am, you’ll love this burger stand.

However, despite the amazing food, epic portions and fantastic service, Burger Bar operates on trade secrets. The recipes for their burger sauces, dipping sauces and shakes are all proprietary. Further, they aren’t free. I pay ~10-12 dollars for lunch whenever I want to pay them a visit. If I bring a party of 6 or 8, I don’t get a bulk discount either. So, aside from the food and the service, everything about the experience is proprietary and vendor-controlled.

I’m okay with that. So why is it that some people aren’t? Well, not with burgers, but with SaaS, or “software as a service”. Of course, I’m referring to Facebook, Google+, Gmail, Bit.ly, and other software vendors that provide an online service to their userbase.

It seems to be the latest “fad” (call it what you will) to oppose proprietary SaaS solutions, or sites with proprietary JavaScript licenses. Companies, such as Facebook, operate on trade secrets. Their server-side code certainly isn’t open to the public, and their JavaScript is obfuscated as much as possible to prevent prying eyes from making any sense out of it (as well as minimize bandwidth). Now, I no longer have a Facebook account, but I left Facebook for other reasons. Mostly, if Facebook was a burger joint, I’m confident that they are trying to poison me, without me catching on. But that’s beside the point. Facebook offers a service, entirely proprietary, much the same way Burger Bar offers a service, entirely proprietary.

Yet, it’s okay to eat the burger, but not okay to use Facebook. It’s okay to ignore the trade secrets of a restaurant, but not okay to ignore the trade secrets of a software vendor. Now, don’t get me wrong. I’m certainly not advocating, endorsing or condoning trade secrets, such as copyrights, patents, trademarks, etc., where the intent is to defend your intellectual property at all costs. All I’m saying is, when it comes to software, I view SaaS a bit differently than installed software.

Continuing the food analogy, when I prepare food in my home, I want to know what’s in it. The FDA in the United States feels the same, and as a result, ingredient lists are required to be printed on every packaged food source. So, when making my own burger, I have the right to know exactly how to prepare it, down to making my own “secret sauce”. I have the source code to my burger, so to speak, and I can make all sorts of fantastic burgers with that “source”. Yet, when I visit a restaurant, I don’t need to know the “source code”, so long as I feel confident the restaurant isn’t trying to poison me or make me sick.

I treat my computer much the same way. My laptop is my home, where I can make my own recipes to create my own software. I have full control over my data, and by having access to the source, make sure the software is respecting my data too (among other things). Google is my restaurant, where I can order software, perhaps pay a premium, and enjoy a good experience, with someone else’s trade secrets. I decide what data to give them, and what not to. I still have full control over my data. So, although I don’t have access to the source, I don’t have to give them my Social Security Number either. On my laptop, having access to the source code is key, and the foundation for a lot of my Free Software principles. On a web site, regardless of the site, I’m not interested in the source code so much, as I am having a positive experience that allows me to interact in a safe and productive manner.

I share this post, because I just finished reading http://ebb.org/bkuhn/blog/2011/11/24/google-plus.html. Bradley Kuhn argues that you won’t find him on these services, such as Twitter or Facebook, because of the trade secrets. I applaud him for sticking to his principles, and not compromising. However, does he eat at burger joints where trade secrets have been critical to their success? I’m curious where the line is drawn. Why is it okay to eat and physically digest trade secrets, but it’s not okay to execute them in your browser? As a result, I believe Bradley may be distancing himself from those that love him, and just want to interact with him online. In fact, I would say he’s distancing himself from the very people he wants to advocate to. How can more people use Free Software, if you are only hanging out with the people who already do, and you are not hanging out with the people who don’t?

Just my thoughts. I’m not interested in trolling, so don’t take this article as such. Only as discussing an angle to SaaS that I don’t think many have thought about. If you’re interested in arguing in the comments, please be civil. Thanks.

2011 11 28 OSS Comments (5) Shortlink

Tab Completing Aliases For Multiple Accounts In Mutt

In mutt, you can setup multiple accounts, and use account hooks (complete with key bindings) to change to those accounts. I have a Gmail account and a work account. In my Gmail account, I use goobook to access my Google Contacts, and I can successfully tab-complete the addresses when composing mail. But, I have not been able to tab-complete my aliases for my work account. Well, I figured it out, and if this is bothering you, here’s the solution:

In my ~/.muttrc:

folder-hook "gmail.com" "source ~/.mutt/gmail.rc"
folder-hook "example.com" "source ~/.mutt/work.rc"
source ~/.mutt/gmail.rc # open gmail on startup

In my ~/.mutt/gmail.rc:

bind editor <Tab> complete-query
bind editor ^T complete
set query_command = "goobook query '%s'"

In my ~/.mutt/work.rc:

bind editor <Tab> complete        # default Mutt setting
bind editor ^T complete-query     # default Mutt setting
unset query_command               # default Mutt setting
source ~/.mutt/work_aliases

Notice the differences between the key bindings for “complete” and “complete-query” in the different RC files. Also notice that I’m unsetting query_command in my work.rc config. This is necessary to tab-complete the aliases out of the ~/.mutt/work_aliases file (the account details for my Google Account reside in ~/.netrc).

Hope this is helpful to someone else. I’m sure this is only helpful for a very small subset of users, but I wouldn’t be doing my due diligence if I didn’t post it. https://www.xkcd.com/979/ is relevant.

2011 11 22 General Comments (4) Shortlink

Unknown Scheduled Downtime

Someone is purchasing our house, and we have to be out by the 28th of November. We will not be in our new house until Dec 3rd, at the earliest. During that week, I don’t know where to host my server to maintain a constant connection. Hopefully, I can find a solution, but worst case scenario, it will be down that entire week. I hope not, but heads up just in case.

Thanks, and sorry for any inconvenience.

2011 11 19 Personal Comments (1) Shortlink

Use QR Codes For Accessing Wireless Access Points

If you have an Android device with a camera, you can install the ZXing Barcode scanner [1] which supports the following method. It is my understanding, however, that other barcode scanners do not support this, so as cool as this is, it may only serve a very limited audience. The ZXing app doesn’t even support this method on iOS, as far as I know.

The ZXing team has a proposal for scanning barcodes to access wireless access points using a MECARD-like structure [2]. The structure of the format is like this:

WIFI:T:WPA;S:mynetwork;P:mypass;;

The parameter “T” can be one of “nopass”, “WEP” or “WPA” for the security type. The parameter “S” is your wireless access point’s SSID (make sure you append “_nomap” to prevent Google from tracking you [3]). The parameter “P” is the password, if any, of accessing the access point. So, a QR Code containing this information could be something like:

Hopefully other barcode scanners pick up on the proposed standard, and make this more ubiquitous. The obvious advantage is not having to type in lengthy passwords on a small screen. At any event, hope this is useful for some.

1: https://code.google.com/p/zxing/
2: https://code.google.com/p/zxing/wiki/BarcodeContents#Wifi_Network_config_%28Android%29
3: http://pthree.org/2011/11/15/google-wants-to-track-your-physical-location/

2011 11 15 General Comments (4) Shortlink

Google Wants To Track Your Physical Location

From http://googleblog.blogspot.com/2011/11/greater-choice-for-wireless-access.html:

We’re introducing a method that lets you opt out of having your wireless access point included in the Google Location Server. To opt out, visit your access point’s settings and change the wireless network name (or SSID) so that it ends with “_nomap.” For example, if your SSID is “Network,” you’d need to change it to “Network_nomap.”

Great. Just great. Google will now be tracking my wireless access point unless I append “_nomap” to the SSID. How many people do you think are going to do this? How many people have even changed their default AP login from “admin:admin”? Google is taking advantage of people, and they know it. I hope the backlash is severe, because I find this to be a breach of trust. Whatever happened to “Do No Evil”?

2011 11 15 Security Comments (20) Shortlink

Rejected And Legal

Some of the roles I fill at work are: storage architecture, cloud engineering, system administration and backend coding. When approaching my tasks head on, it’s always important to me that standards are adhered to. From PEP coding style to adhering to an RFC for mail server. Unfortunately, I think I’m a dying breed, or something, because more and more, I’m seeing standards ignored.

Case in point: I just filled out a form for a survey to “enter to win a $1000 shopping spree). You know, the crap that you constantly get bombarded with at the checkout stand when the cashier gives you your receipt. I always ignore them, but then thought to myself “I’ll never win if I don’t at least try”, so I gave my first survey a go. At the end of the survey, it asked for my email address. I figure they’ll sell it for marketing purposes, and I have a Google Mail address, so I’m not really that worried about the SPAM (their SPAM filters are amazing). But, I would like to track who they are selling my address to. So, I gave them the following address:

aaron.toponce+survey-provider@gmail.com

To which, I received an error that the email address is not a valid address. AHEM! Yes it is, and it’s this lack of support for standards that I’m talking about. My email address was rejected, yet it’s perfectly legal according to RFC 5322. You see, according to that RFC, I get the following flexibilities with my email address:

  • ASCII upper and lower case letters (a-z & A-Z).
  • ASCII digits 0-9
  • ASCII characters !#$%&’*+-/=?^_`{|}~
  • ASCII dot (.) so long as the local part of the address does not contain the dot consecutively, and it does not start with a dot.
  • ASCII characters ” ” (space) and “(),:;<>@[\] are allowed with certain restrictions.

So, I could have the following email addresses, all of which are perfectly legit according to the RFC:

  • “[Aaron Toponce]“@gmail.com
  • a&t@gmail.com
  • aaron.toponce+business@gmail.com
  • aaron’s-travel-agency@example.travel
  • {atoponce}@gmail.com

Yet, these will get ejected outright in most web forms I’ve come across. Specifically interesting is the .travel TLD. I’ve had web forms enforce TLDs that are less than 4 characters, which is absolutely absurd for the .travel and .museum TLDs. I’m guessing one of two things is happening with these web forms:

  1. The developer used the regular expression [A-Za-z0-9_\-\.]+@[A-Za-z0-9\-\.]+ for validating addresses.
  2. There is absolute denial for the use of “plus-addressing” as a DEA.

I’m guessing the first is more likely the scenario than the second. Regardless, Of course, when we’re talking about the rules of RFC 5322, we’re no longer talking about regular expression syntax. We’re talking about grammar. If your page is designed in PHP, Python, CGI, or whatever, you should use a real parser for parsing the email address, rather than reinventing the wheel yourself. What’s unfortunate, is this disease of not properly parsing valid email addresses is found in some big companies and sites too, not just the little guys.

Now, Google COULD provide true DEAs, such as Yahoo! Plus does with their subscribers, However, I should be able to create an DEA with an already existing email address, rather than creating completely new ones, because people refuse to conform to the standards. So Google, if you’re reading (I know you are), you may want to consider proper DEAs, seeing as though “plus addressing” isn’t working, and it is important to some.

Stéphane Bortzmeyer has already blogged about this, and he uses the #ral hashtag on Identica and Twitter to vent his frustrations, which stands for “Refus d’Adresses Légales” or “Rejection of Legal Address”. Well, I’ve determined that I will be doing the same, although I’ll bacronym the hashtag to “Rejected And Legal”, along with the url to the site that refuses to adhere to the RFC.

2011 11 10 Scripting Comments (5) Shortlink

Use wget(1) To Expand Shortened URLs

I’m a fan of all things microblogging, but let’s face it: until URLs become part of the XML, and not part of your character count (which is ridiculous anyway), shortened URLs are going to be a way of life. Unfortunately, those shortened URLs can be problematic. They could host malicious scripts and/or software that could infect your browser and/or system. They could lead you to an inappropriate site, or just something you don’t want to see. And because these URLs are a part of our microblogging lives, they’ve also become a part of our email, SMS, IM, IRC, lives as well as other online aspects.

So, the question is: do you trust the short URL? Well, I’ve generally gotten into the habit of asking people to expand the shortened url for me if on IRC, email or IM, and it’s worked just fine. But, I got curious if there was a way to do it automagically, and thankfully, you can use wget(1) for this very purpose. Here’s a “quick and dirty” approach to expanding shortened URLs (emphasis mine):

$ wget --max-redirect=0 -O - http://t.co/LDWqmtDM
--2011-10-18 07:59:53--  http://t.co/LDWqmtDM
Resolving t.co (t.co)... 199.59.148.12
Connecting to t.co (t.co)|199.59.148.12|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://is.gd/jAdSZ3 [following]
0 redirections exceeded.

So, in this case “http://t.co/LDWqmtDM” is pointing to “http://is.gd/jAdSZ3″, another shortened URL (thank you Twitter for shortening what is already short (other services are doing this too, and it’s annoying- I’m looking at you StatusNet)). So, let’s increase our “–max-redirect” (again, emphasis mine):

$ wget --max-redirect=1 -O - http://t.co/LDWqmtDM
--2011-10-18 08:02:12--  http://t.co/LDWqmtDM
Resolving t.co (t.co)... 199.59.148.12
Connecting to t.co (t.co)|199.59.148.12|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://is.gd/jAdSZ3 [following]
--2011-10-18 08:02:13--  http://is.gd/jAdSZ3
Resolving is.gd (is.gd)... 89.200.143.50
Connecting to is.gd (is.gd)|89.200.143.50|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://wiki.ubuntu.com/UbuntuOpenWeek [following]
1 redirections exceeded.

So, in this case, the link finally points to https://wiki.ubuntu.com/UbuntuOpenWeek. I’m familiar enough with the Ubuntu Wiki, that I know I should be safe visiting the initial shortened URL. If you want to add this to a script or shell function, then you can get a bit more fancy:

$ expandurl() { wget -O - --max-redirect=$2 $1 2>&1 | grep ^Location; }
$ expandurl http://t.co/LDWqmtDM 1
Location: http://is.gd/jAdSZ3 [following]
Location: https://wiki.ubuntu.com/UbuntuOpenWeek [following]

In this case, our “expandurl()” function takes two arguments: the first being the URL you wish to expand, and the second being the max redirects. You’ll notice further that I added “-0 -” to print to STDERR. This is just in case you give too many redirects, it will print the content of the page’s HTML to the terminal, rather than saving to a file. Because you’re grepping for “^Location”, and sending the HTML to your terminal anyway, technically you could get rid of the “–max-redirects” altogether. But, keeping it in play does seriously increase the time it takes to get the locations. Whatever works for you.

UPDATE (Oct 18, 2011): After some comments have come in on the post, and some discussion on IRC, there is a better way to handle this. According to the wget(1) manpage, “-S” or “–server-response” will print the headers and responses printed by the FTP/HTTP servers. So, here’s the updated function that you might find to be less chatty, and faster to execute as well:

$ expandurl() { wget -S $1 2>&1 | grep ^Location; }
$ expandurl http://t.co/LDWqmtDM
Location: http://is.gd/jAdSZ3 [following]
Location: https://wiki.ubuntu.com/UbuntuOpenWeek [following]

Perfect.

2011 10 18 Scripting
Security Comments (12) Shortlink

Why Firefox Is My Browser Of Choice

Posted to my Google Plus account (https://plus.google.com/115784859563110525602/posts/cwwMBdh4iPL):

Let’s talk browsers. Specifically, Opera 11.51, Firefox 7.0.1, Safari 5.1 and Chrome 14.0.835.163. I don’t have access to Internet Explorer 9. Ordered best to worst.

JavaScript standards compliance: http://test262.ecmascript.org (lower failures is better):
Firefox: 191 failures
Chrome: 425 failures
Safari: 832 failures
Opera: 3,750 failures

HTML5 standards compliance: http://html5test.com (higher is better):
Firefox: 313 and 9 bonus points
Safari: 293 and 8 bonus points
Opera: 286 with 7 bonus points
Chrome: 284 and 2 bonus points

CSS3 standards compliance: http://acid3.acidtests.org (higher is better):
All score 100% with fluid animation.

JavaScript benchmark performance: http://webkit.org/perf/sunspider/sunspider.html (lower is better):
Firefox: 211.9ms +/- 1.3%
Opera: 258.2ms +/- 2.7%
Safari: 291.4ms +/- 0.9%
Chrome: 298.3ms +/- 7.1%

In every test, Firefox 7 came out on top. I realize it’s the newest browser of the bunch, but it’s fairly clear that it’s holding its own against its competitors. It performs, it’s standards compliant, and it’s still my browser of choice. Further, it’s interesting that Firefox beats the pants out of Chrome in 3 of the 4 tests, and yet Chrome is gaining market share. Go figure.

2011 09 30 Firefox Comments (27) Shortlink

Salt Packaged For Debian/Ubuntu

I created a Debian package for Salt, doing my best to adhere to the Debian Policy. You can find it at http://mentors.debian.net/package/salt. If you’re not familiar with Salt, head over to http://saltstack.org and check it out. From the author’s website:

Salt is a powerful remote execution manager that can be used to administer servers in a fast and efficient way.

Salt allows commands to be executed across large groups of servers. This means systems can be easily managed, but data can also be easily gathered. Quick introspection into running systems becomes a reality.

Remote execution is usually used to set up a certain state on a remote system. Salt addresses this problem as well, the salt state system uses salt state files to define the state a server needs to be in.

Between the remote execution system, and state management Salt addresses the backbone of cloud and data center management.

Think Puppet on steroids. And done correctly.

At any event, yes, I’m looking for a Debian Developer to sponsor me getting it into Debian proper. If you are an Ubuntu Developer, and could help sponsor me getting it into Ubuntu as well, that would be awesome!

2011 09 29 Debian
Ubuntu Comments (2) Shortlink

How To Properly Create And Burn CD/DVD ISO Images From The Command Line

Too often, I see the recommendation on forums, IRC, and elsewhere across the internet to use improper tools for creating ISO images. For example, consider the following two commands, both of which are not the correct way to build a CD/DVD image:

$ dd if=/dev/scd0 of=cdimage.iso # NO!

Or worse yet:

$ cat /dev/scd0 > cdimage.iso # NO!

As you are probably thinking, the problem with the two commands above, is that they provide no error checking while building the image. So, in order to make sure you have all the bits, you need to use another tool, such as using the MD5 hashing algorithm:

$ md5sum /dev/scd0 cdimage.iso
d642d524dd2187834a418710001bbf82  /dev/cdrom
d642d524dd2187834a418710001bbf82  cdimage.iso

Thankfully, the hashes above match. But, what if they didn’t? Then, you get to redo your dd(1) command (or, shudder, cat(1)) from above, and then rerun md5sum(1) to make sure you got all the bits. Doesn’t sound like much fun to me. Thankfully, there is a better way, one which handles the checksum while doing the copy.

You want to use readom(1) (“read optical media”) from the wodim(1) (“write optical disk media”) package. Consider the following command:

$ readom dev=/dev/scd0 f=cdimage.iso # YES!

If readom(1) fails to get the bits during the copy, it will let you know that it’s struggling. If it got all the bits, you know you have them all, because of the error checking during the copy. Sure will save you a lot of time running manual hashes when finished.

Now, what about burning a copy of the ISO image? Surely you use dd(1), yes? Something like:

$ dd if=cdimage.iso of=/dev/scd0 # NO!

NO! Instead, use wodim(1) directly:

$ wodim -v -eject cdimage.iso # YES!

For the same reasons that you want to use readom(1) for creating ISO images from CD/DVD, you want to use wodim(1) for burning ISO images to CD/DVD. What happens if after using dd(1) to create your CD/DVD, the md5sum(1) hash doesn’t line up with the image? You didn’t get all the bits, and created a coaster. Use wodim(1) and should it succeed, you can rest assured that you have all the bits.

So, remember, readom(1) and wodim(1) are the tools you want when creating and/or burning ISO images from the command line. Any other tool, and you’re likely doing it wrong.

2011 09 26 Debian
Scripting
Ubuntu Comments (3) Shortlink

Avoid Using which(1)

This post comes from BashFAQ/081 on Greg’s Wiki. He argues why you should not be using which(1) to determine if a command is in your $PATH at the end of the page. I’ll put that argument at the front:

The command which(1) (which is often a csh script, although sometimes a compiled binary) is not reliable for this purpose. which(1) may not set a useful exit code, and it may not even write errors to stderr. Therefore, in order to have a prayer of successfully using it, one must parse its output (wherever that output may be written).

Note that which(1)’s output when a command is not found is not consistent across platforms. On HP-UX 10.20, for example, it prints “no qwerty in /path /path /path …”; on OpenBSD 4.1, it prints “qwerty: Command not found.”; on Debian (3.1 through 5.0 at least) and SuSE, it prints nothing at all; on Red Hat 5.2, it prints “which: no qwerty in (/path:/path:…)”; on Red Hat 6.2, it writes the same message, but on standard error instead of standard output; and on Gentoo, it writes something on stderr.

(Quotation and manpage reference additions mine). So, if which(1) is bad news, then what is the “proper” way to determine if a command is in your $PATH? Well POSIX has an answer, and not surprisingly, the command to use is “command”:

1
2
3
4
5
6
# POSIX
if command -v qwerty >/dev/null; then
  echo qwerty exists
else
  echo qwerty does not exist
fi

The “command” built-in also returns true for shell built-ins. If you absolutely must check only PATH, the only POSIX way is to iterate over it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# POSIX
IsInPath ()
(
  [ $# -eq 1 ] && [ "$1" ] || return 2
  set -f; IFS=:
  for dir in $PATH; do
    [ -z "$dir" ] && dir=. # Legacy behaviour
    [ -x "$dir/$1" ] && return
  done
  return 1
)

if IsInPath qwerty; then
  echo qwerty exists
else
  echo qwerty does not exist
fi

There are also Bash built-ins that can be used, should you have Bash installed on your system:

1
2
3
4
5
6
# Bash using the 'hash' built-in
if hash qwerty 2>/dev/null; then
  echo qwerty exists
else
  echo qwerty does not exist
fi

Or:

1
2
3
4
5
6
7
# Bash using the 'type' built-in
# type -P forces a PATH search, skipping builtins and so on
if type -P qwerty >/dev/null; then
  echo qwerty exists
else
  echo qwerty does not exist
fi

If you prefer the ZSH (my addition not present in the wiki), as I do, then you can look in the $commands associative array:

1
2
3
4
5
6
# ZSH using the $commands associative array
if [[ $commands[qwerty] >/dev/null ]]; then
    echo qwerty exists
else
    echo qwerty does not exist
fi

I like that at the end of the FAQ, he gives a shell script for using which(1) should it be absolutely necessary. Not only do you have to test for exit code, but you also have to test for common strings in the output, seeing as though which(1) doesn’t always use exit codes properly:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# Bourne.  Last resort -- using which(1)
tmpval=`LC_ALL=C which qwerty 2>&1`
if test $rc -ne 0; then
  # FOR NOW, we'll assume that if this machine's which(1) sets a nonzero
  # exit status, that it actually failed.  I've yet to see any case where
  # which(1) sets an erroneous failure -- just erroneous "successes".
  echo "qwerty is not installed.  Please install it."

else
    # which returned 0, but that doesn't mean it succeeded.  Look for known error strings.
    case "$tmpval" in
      *no\ *\ in\ *|*not\ found*|'')
        echo "qwerty is not installed.  Please install it."
        ;;
      *)
        echo "Congratulations -- it seems you have qwerty (in $tmpval)."
        ;;
    esac
fi

CONCLUSION:
You have many options to find whether or not a command exists in your $PATH, some POSIX, some proper built-ins. Regardless, you should be able to build platform-independent scripts using the proper tools, and using which(1) is not the right tool for the job. Hopefully, this has convinced you of that.

2011 09 26 Debian
Scripting
Ubuntu Comments (3) Shortlink