I guess when you have GNU developing both the bash shell and the text editor emacs, you're going to have one set of keyboard shortcuts work in the other. Case in point: the emacs keyboard shortcuts are also bound in the bash shell.
Well, being a vim enthusiast, that just isn't going to work. So, here's something that I learned at my Ubuntu-Utah LoCo meeting that ended just 20 minutes ago:
aaron@hercules:~$ set -o vi
Oh yeah. That's better. When executed, you are in insert mode, and not command mode. So, to take advantage of the vim keyboard shortcuts, 'esc' or 'ctrl-[' to get into command mode, and hack away! Make sure to add this to your ~/.bashrc.