Increase the speed at which the 'delete' key deletes things on OSX?

Under System Preferences, click Keyboard, and adjust the "Key Repeat" speed (see below). Tested in Word and it does affect how quickly the delete key deletes characters when held down.

Alternatively, in the terminal you can enter

defaults write NSGlobalDomain KeyRepeat -int 0

which is the fastest setting. The fastest exposed by System Preferences is 2 (log out and back in to see changes). See here: Mac OSX Hints

Additionally,

defaults write NSGlobalDomain InitialKeyRepeat -int 15

can be used to change the delay before the repeat starts (default is 15).

For a graphical solution, see keyremap4macbook, detailed here.

You can also hold the Option key while pressing delete. This deletes whole words instead of characters. Similar functionality to holding Control while pressing backspace in windows.