When a key is pressed on the keyboard, a number of hardware and software components cooperate so as to guarantee that the intended meaning of the key (e.g., emitting a certain character) matches the actual behaviour of the key. I will concentrate on the software side (as our control on the hardware part is nonexistent), and in particular, for the time being, on the events related to console output.
Hitting a key causes raw keyboard scancodes to be generated; these scancodes are then transformed in a keycode. On an i386 system, usually the key Backspace emits 14 and the key Delete emits 111.
The keycodes are translated by the keyboard library into a keyboard symbol (keysym) using the keyboard definition loaded by the user. If you look into your keyboard database (e.g., in /lib/kbd/), you'll discover several definitions for different computers, different layouts and possibly different interpretations of the same keys (e.g., one could desire that the two Alt keys really behave as distinct modifiers). The Linux console keyboard layout assigns keysym Delete to keycode 14 and keysym Remove to keycode 111. This may seem strange, but the Linux console emulates a VT100 terminal, and this is the way things work in that realm.
Our journey has still to come to an end. Console applications read ASCII sequences, not keysyms. So the console must read keysyms and translate them into ASCII sequences that suitably encode the keys. Of course, this operation must be performed in a way that is understandable by applications. For instance, on the Linux console the Delete keysym is mapped to the ASCII code 127 (DEL), the Remove keysym on a suitable escape sequence, and the BackSpace keysym to ASCII code 8 (BS).
Finally, we must in a sense roll back to what we had before and translate the ASCII sequences generated by each key into a key capability. This goal is reached by a terminal database, which contains, for each kind of terminal, a reverse mapping from sequences of characters to key capabilities (which are essentially a subset of the keysyms).
Unfortunately, there are two "standard" terminal databases, termcap and terminfo. Depending on your distribution, you could be using either one of them, or the database could even depend on the application. Our discussion will concentrate on the more modern terminfo database, but the suggested fixes take both into consideration.
For instance, on the Linux console F1 generates an escape followed by [[A, which can be translated to the capability key_f1 by looking into the terminal-database entry of the console (try infocmp linux if you want to have a look at the entry). A very good and thorough discussion of terminal databases can be found in GNU's termcap manual. Usually Linux applications use the newer terminfo database, contained in the ncurses package.
Maybe at this point not surprisingly, the Linux console terminfo entry maps DEL to the kbs (backspace key) capability, and escape followed by [3~ to the kdch1 ("delete-one-char" key) capability. Even if you could find strange that the Backspace key emits a DEL, the terminal database puts everything back into its right place, and correctly behaving applications will interpret DEL as the capability kbs, thus deleting the character to the left of the cursor.
This claim has been asserted/disputed several times commenting this document. If you have any definitive information on this subject, please write me.
Some programs rely on the terminal driver for input line editing, such as deleting characters or words. With stty, you can tell the terminal driver what character it should use to delete the character to the left of the cursor (the erase character). You can check your current settings with stty -a and set them with stty erase character.