Now, I’m no Git expert, that’s for sure. So, I won’t be getting into the reasons on why you should or should not setup a VCS, nor will I cover the more intermediate or advanced features of Git. However, setting up a Git repository is easy, and I’ll show how you can manage your own config files using Git.

The latest versions of Debian and Ubuntu, as well as Fedora and openSUSE, all have Git available in their repositories. Third-party packages are available for RHEL, and you can grab a package for SLES at their build service.

Ok, so let’s get Git installed. I’m going to use Git through an SSH server, which means you’ll need to have an already installed SSH server that you can remotely access. Once that is setup, then we will need to run the following on both our remote SSH server, and our local machine. I’ll refer to the remote SSH server as “remote”, the local box as “local” and both machines as “both”. Also, I’ll use the “#” for root commands, and “$” for regular user commands. So, on my Ubuntu systems as root:

both# aptitude install git-core git-doc

You should always install the documentation package, if it exists, when dealing with new software. In this case, the Git developers have done a great job in this area, so you can be more effective in your development.

After installed, we need to setup the repositories. One of the selling features for me with Git, is the fact that a Git repository s 100% self-contained. This means, that if you make some flaw setting it up, or it becomes severely corrupted and you can restore from backup, then all you need to remove is the .git directory recursively, and your repository is gone. No fiddling with config files throughout the system, or running obscure commands. When a Git repository is setup, it creates a .git directory, and it’s all managed there. So, on the remote machine, let’s create a repository in our home directory, where our central repository will be managed:

remote$ mkdir ~/my_repo
remote$ cd $_
remote$ git init
Initialized empty Git repository in /home/aaron/my_repo/.git/

Now, let’s do the same thing on our local machine. However, I generally deal with source code in my ~/src directory, so that’s where I’ll put my configs:

local$ mkdir ~/src
local$ cd $_
local$ git init
Initialized empty Git repository in /home/aaron/src/.git/

We have now created two empty Git repositories- one on the remote machine behind SSH, and the other on our local machine. The unique thing with Git, is that it doesn’t follow the standard “server/client” architecture. In other words, there’s no such thing as a “git server” and “git client”. Git was developed by filesystem developers with filesystem attributes in mind. So, instead, we have a remote Git repository we call the “origin” and a local Git repository (I’m not sure if it has an appropriate name. Maybe “git trunk”?). Both are contained in the .git directory. The only difference is, one will be holding the files and directories, while the other will be managing deltas on the files and directories.

So, let’s continue. On the local machine now, I need to start doing my developing. Because my repository is keeping track of local dot config files in my home directory, I’m going to copy my .zshrc file to the Git repository on the local machine:

local$ cp ~/.zshrc ~/src
local$ cd $_
local$ git status
# On branch master
#
# Initial commit
#
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
#       .zshrc
nothing added to commit but untracked files present (use "git add" to track)
local$ git add .zshrc
local$ git status
# On branch master
#
# Initial commit
#
# Changes to be committed:
#   (use "git rm --cached ..." to unstage)
#
#       new file: .zshrc
#
local$ git commit -a -m "initial commit for .zshrc"
Created initial commit 0614613: initial commit for .zshrc
 1 files changed, 23 insertions(+), 0 deletions(-)
 create mode 100644 .zshrc
local$ git status
# On branch master
nothing to commit (working directory clean)

So, if you paid close attention, all we’ve done so far is copy my .zshrc from my home directory into my src directory. It’s best practice to get into the habit of running “git status” and “git commit” often. This will prevent any future headache you may encounter. So, upon initial discovery, I find out that I need to add my newly copied .zshrc file. After “git add .zshrc”, I then need to commit it to the branch I’m working on, which in this case is called “master”. As it sits now, the status of my Git repository is clean. So, all I need to do now, is push it to the remote system:

local$ git push ssh://myserver.com/~/my_repo master
Counting objects: 3, done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 671 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
To ssh://myserver.com/~/my_repo
 * [new branch]      master -> master

Seeing “master -> master” is a good sign. This is telling me the push was successful, and now my remote SSH server has a copy of my repository. Now, to show that it does in fact exist, let’s remove our .git directory, and clone it;

local$ cd ~
local$ rm -rf src
local$ mkdir src
local$ cd $_
local$ git clone ssh://myserver.com/~/my_repo
Initialized empty Git repository in /home/aaron/src/my_repo/.git/
remote: Counting objects: 3, done.
remote: Compressing objects: 100% (2/2), done.
Receiving objects: 100% (3/3), 671 bytes, done.
remote: Total 3 (delta 0), reused 0 (delta 0)
local$ ls -a
. .. my_repo
lcoal$ ls -a my_repo
. .. .git .zshrc

Further, check out our ~/.git/config file on our local machine:

local$ cat ~/src/my_repo/.git/config
[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[remote "origin"]
	url = ssh://myserver.com/~/my_repo
	fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
	remote = origin
	merge = refs/heads/master

Notice the “remote origin” line. The url is set to our server that we cloned from. This means, that when I want to push my local repository changes to the remote server, I can execute the following instead:

local$ git push origin master

So there it is: setting up a simple Git repository for managing config files. No need for Gitosis, or other “applications” to make it possible. However, this was a simple demonstration, and doesn’t cover allowed users to work on the repository, who can push and who can’t and so forth. But, it does get us started on the right foot.

Happy Git hacking!

First, we need to get a good grasp of screen. Screen is what we call a “terminal multiplexer”. A user attaches himself to a console on the machine, then at that users whim, detach from the console, but still leave it installed and its attached processes running on the system. What makes screen so attractive is the ability to reattach to that console, with its currently running processes available. Generally, screen is run on a remote SSH server, so the administrator can log into that box from anywhere, reattach his running screen session, and continue working on his applications like he never left the machine.

Heres how it works. Log into your local machine, and type:

$ who am i
aaron    pts/1        2008-10-12 05:43 (:0.0)

This shows me logged in as aaron, and attached to pseudo terminal 1. I logged into the system on October 12, 2008 at 5:43 in the morning (my daughter won’t sleep). Now, let’s execute screen, and run a couple of commands:

$ screen
$ who am i
aaron    pts/4        2008-10-12 05:45 (:0:S.1)
$ cat /dev/zero > /dev/null &

Still logged in as aaron, but this time, I’m attached to pseudo terminal 4. Also, we ran a pretty worthless command, however, it will illustrate the point. Detach from your screen session by typing the following keyboard sequence:

ctrl-a d

This will “detach” you from pseudo terminal 4, but still leave the device created, will all running processes attached to it:

[detached]
$ ls /dev/pts/4
/dev/pts/4
$ ps -ef | grep cat
aaron    27684 27494 93 05:49 pts/4    00:00:05 cat /dev/zero

The process that we ran in screen a moment ago is still running. In my case, it has a process ID of 27684 with a parent process ID of 27494 and attached to pseudo terminal 4, just like we would expect. I wonder what process ID 27494 is…

$ pstree -p | grep -B 1 27494
        |-screen(27492)-+-ssh(27493)
        |               `-zsh(27494)---cat(27684)

(The ‘pstree’ command, if run on its own, will show your the processes running on your box in a tree-like structure, showing parent and child processes).

As we can clearly see, ‘screen’ is process ID 27492, with a child process of ZSH, my favorite shell, running as process ID 27494. ZSH is currently running our ‘cat’ command as process ID 27684, as we just discovered.

Here’s what is absolutely great about GNU screen: I can detach from sessions and reattach to those sessions at will. Suppose I ran that command on my box at home. When I get to work, I want to keep an eye on it. All I need to do now, is login remotely to my machine (hopefully, it’s running an SSH server), and reattach my screen session:

% ssh helios.cocyt.us
Password:
$ who am i
aaron    pts/0        2008-10-12 06:00 (hades.cocyt.us)
$ screen -r
$ who am i
aaron    pts/4        2008-10-12 06:01 (:0:S.1)
$ kill 27494
[1]  + terminated  cat /dev/zero > /dev/null
$ exit
[screen is terminating]

Not only can we detach and attach ourselves from and to these processes respectfully, we can also create new terminals in screen, split our screen horizontally, add multiuser sessions, lock the screen from intrusion, and many other features. The capabilities of screen go well beyond what we’ve discussed here. Suffice it to say that GNU screen is one of the most powerful tools that a system administrator can have in his tool belt.

GNU screen is available in the Fedora, openSUSE and Ubuntu repositories, as well as Red Hat Enterprise Linux and SUSE Linux Enterprise Server and Desktop.

Before I begin, however, let make take a couple seconds to explain what this post is and is not about. This post is all about learning about shadowed passwords on GNU/Linux. This post is not about learning how to implement different shadowed algorithms on your distribution, the pros and cons of each algorithm, or the mathematical details behind them. This post serves mainly as an informative post regarding the shadowed password itself.

Also, I need to declare a couple of definitions:

• shadowed password: A password stored in a database that is only readable by the root user on a Unix or Linux machine. Unix didn’t implement shadowed passwords until 1988 with AT&T SystemV R3.2 and BSD 4.3 Reno in 1990.
• salt: A random string of text that is used as a seed to build an encrypted hash. Passwords on Unix and Linux are hashed with this salt as a seed. The salt is passed to the hashing algorithm as an argument.

Let’s familiarize ourselves with the entire syntax, start to end, of the /etc/shadow file. Then, we’ll get dirty digging into the nitty-gritty of the actual password, and how authentication on the machine works. So, first the overall syntax.

As you’ve probably noticed, the /etc/passwd and /etc/shadow as well as /etc/group and /etc/shadow (if that exists) are colon-delimited files. This means, that columns exist by using the colon as a separator, exactly the same way commas would describe columns in a comma separated file. Between each column, in the /etc/shadow file, exists a value describing aging information regarding the account. Let’s look at it in more detail:

username:password:last_change:may_change:must_change:warn_days:expire_date:disable:reserved

Looking at each of these fields one by one:

• username: The username of the account as specified in the /etc/passwd file
• password: The password is in encrypted hash format, meaning that it has been hashed using a salt as the seed. The rest of the post is on this very topic.
• last_change: The number of days, since January 1, 1970, when the password was last changed.
• may_change: The number of days since the last password change before a new password change is allowed.
• must_change: The number of days since the last password change when a new password change is required.
• warn_days: The number of days before must_change when the user begins receiving warning about changing their password.
• expire_date: The number of days after the password expires when the account is disabled.
• disable: The number of days since January 1, 1970, when the account will be disabled.
• reserved: A reserved field for the system.

All of this information can be found in section 5 of the shadow man page.

Knowing the locations and valid values of each is crucial as a system administrator. However, I won’t be covering each of these in any detail other than what has been described above. Also, I’ll only be discussing the password field. The password field is interesting to me, as I enjoy mathematics and cryptography. Hopefully, you’ll find this interesting as well.

As you are probably already aware, the password field is required in order to use the account, however, a password doesn’t have to exist for the account to be active. When a new account is created on the system, before a password is applied, take a look at the contents of the password in the shadow file. On my Ubuntu system, there is a single exclamation point in the password field. As it sits, the password is invalid, as the system is looking for a hash. Removing the exclamation point from the password field would mean that the user can log into the system without a password. It should be noted, however, that there are unexpected behaviors logging into the system without a password with regards to PAM properly cleaning up the login.

This brings up another point: the authentication mechanisms that exist on your machine are attempting to validate your password against a valid hash that exists in that password field. Keeping a user from logging in is as simple as producing an invalid hash. This means simply changing a single character in the password, or as above with newly created accounts, putting an exclamation point or two in the password field. In other words, unless a hash can be produced that matches a hash in that field, the user won’t be able to log in. Fairly simple, no?

If you check the permissions of the /etc/shadow file, you’ll notice that only root has access to the file. This is intentional, for one main reason: rainbow table attacks. The concept of a rainbow table is simple. Take some text, retrieve the hash value then store it in a table. As hash algorithms are one way cryptographic functions, this means that taking a hash value, and reversing it to its text value is extremely time consuming and highly improbable. So, rather than brute force the hash value, all you have to do is simply look up a hash in the table for a match. If a match is found, you have the text to the hash. Spending some time on Google, there are plenty of rather large rainbow tables in existence. Salting the hash means that access to the salt would be needed, otherwise, the hash will change infinitely for a single password.

A little history note: passwords have not always existed in the shadow file, as they used to exist in the /etc/passwd file. After all, where do you think the name came from? In the second field of that file is where the password used to exist. Now, you’ll notice that an ‘x’ exists in its place instead, with the password moved to the shadow file. The reason is explained above, with rainbow tables. The /etc/passwd permissions allow any account on the system to read the file. This is needed for applications to know if accounts exists on the system for various routines. Keeping the password, even if it’s hashed, in a file world-readable made system administrators uneasy, and eventually, the shadow system was created. FYI- on FreeBSD, and other Unixes, the shadow file is /etc/master.passwd rather than /etc/shadow.

To bring security up on the front lines, hashes aren’t difficult to brute force. When Unix was created in 1969, computing resources were severely limited. We couldn’t use massive mathematical algorithms to process hashed passwords and see if they matched what was in the password database. As such, we stuck with small bit rates for generating the hash. As time progressed, and computing power got more powerful, the need for changing that bit rate to higher values on creating the hash increased. Unfortunately, the rate of change wasn’t linear to the rate of change in computing power, and brute forcing hashed passwords became more and more of a realization. The need for stronger algorithms became a necessity.

As already noted, the password was stored in the /etc/passwd file. However, because the threat of brute forcing password hashes was becoming real, the need to rethink passwords on a Unix system was eventually realized, and the password moved into its current location, the /etc/shadow file. The reason for “shadow” comes from the idea that rather than just storing the password hash in a world-readable file, we move it to a separate file that only root has access (as defined above). Further, to add strength to the hashed password, we create a seed, which we’ll refer to as a “salt”, and use that salt, combined with the hashing algorithm, to result in a much stronger encrypted password (security experts actually will call the encrypted password encoded rather than encrypted, as the text is set to null, and the password is the key). Shadowing and salting your password has been implemented in GNU/Linux for some time, and came as a result of first appearing in FreeBSD. Most, if not all, GNU/Linux distributions now shadow and salt their passwords by default.

So, now that we’ve learned what is contained in the /etc/shadow file, and a little history of the password itself, let’s look at the structure of the encrypted password in the file. You will probably notice that it has a very patterned format. The format is simple to understand:

$hash_type$random_salt$encrypted_password

There are 3 dollar signs which act as separators in the shadowed password field. Within the first two dollar signs exists the type of algorithm used to hash the password. Within the next two dollar signs is a random salt that is used with the hashing algorithm to create the encrypted password. After the third dollar sign, before the next colon, is the encrypted password itself.

For the supported hash types, this is generally MD5, however, new computing power is beginning to show the age of MD5 hashes, which uses a 128-bit key. As such, new hashing algorithms are coming onto the scene to strengthen the password itself, such as 160-bit, 256-bit and 512-bit algorithms. The question is, however, how do know which algorithm you are using? I’ve outlined a list of the common, currently in use hash types.

• missing: If the dollar signs are missing, but a hash exists, then the DES algorithm or the crypt() function was called. This has been implemented on Unix since the early days, and still exists on a few GNU/Linux distributions. DES has shown severe cryptanalysis attacks, and should not be used.
• $1$: The MD5 hashing algorithm, as originally started with FreeBSD, developed by Poul-Henning Kamp. This is the default on Ubuntu, and other Debian-based distributions. It uses a 128-bit key during creation.
• $2$ and $2a$: Two different versions of the blowfish encryption cipher, originally developed by Bruce Schneier, and implemented by Niels Provos and David Mazieres for OpenBSD. This is the default in openSUSE 11, as well as the forthcoming SUSE Linux Enterprise Server and Desktop, version 11. It also uses a 128-bit key, where $2$ appends a NUL to the key, and $2a$ does not.
• $5$: The next two come from Ulrich Drepper at Red Hat for GNU/Linux. This one is a SHA-256 hashing algorithm implementation with a 32 byte output, and uses a 256-bit key.
• $6$: This is a SHA-512 hashing algorithm implementation with a 64 byte output. This is the default on Fedora 9 and soon to be released Red Hat Enterprise Linux 6. It uses a 512-bit key.

Others exist, but aren’t worth mentioning, in my opinion. However, for reference, there is a CPAN Perl module for working with shadowed passwords, and documentation exists for handling all the supported shadow types.

I won’t be covering the mathematical algorithm used on each function, as that could make for an exceptionally lengthy post (you’re probably bored already). As such, may I recommend spending some time on Google and Wikipedia, learning each algorithm, their strengths and weaknesses, as well as their implementations? It really makes for fascinating reading, if you are into mathematics and cryptography.

Now, what’s up with the salt? What does it actually provide? Why should I care? Isn’t a straight hash strong enough? Actually, no- it’s not. A straight hash, albeit strong, is not strong enough, due to rainbow tables. So, the salt steps in, and adds strength to the hash. Here’s how. First, a string is chosen at random with the following valid characters: a-z, A-Z, 0-9, “.” and “/”. In other words, all 26 uppercase and lowercase letters of the English alphabet, all ten single digits, the period and forward slash.

Let’s say that only 2 characters where chosen from that pick to form our salt string. Because there are 64 possible combinations when choosing just one character, and another 64 combinations for the other, this makes the possible combinations from just 2 characters 4096 (64 squared). Adding that to the hash, means there are 4096+1, or 4097 possible hashes that could result from my single password. This makes the rainbow table 4096 times larger, and it also makes the hash literally 4096 times more difficult to break. Now, on my Ubuntu installation, the salt is used with 8 characters from the above list. So, 64 to the 8th power means 281,474,976,710,656 possible combinations in the salt alone. Which, as discovered above, means 281.4 quadrillion possible hashes from a single password, and a rainbow table size of 281.4 quadrillion times larger! To put this number in perspective, 281,474,976,710,656 seconds would be approximately 8,925,513 years. Yeah. Salting the password just makes sense, don’t you think?

To take advantage of a specific algorithm in the shadow file, we need to modify the pluggable authentication modules, known as PAM. Of course, you will need to have an understanding of PAM before hacking away. I’m not going to cover how to setup your box to take advantage of the different algorithms. The would lengthen the post, and potentially render your system unbootable if you were to make a mistake, forcing you into a rescue environment, then I get blamed. However, check your GNU/Linux vendor documentation, as well as Google, and support forums for enabling different algorithms on your distribution.

So, there you have it! Shadowed password information on GNU/Linux. As you can tell, there is much to discuss in relation to this topic. I could probably spend a couple of posts describing how to implement each algorithm on your box, the algorithms of each, testing their strengths and weaknesses, and much more. Encryption, especially when it comes to passwords, is a broad, but intriguing and enlightening topic.