Delete space-expanded "tab" in Vim with one keystroke?

set softtabstop=4

You can use Ctrl+D to back up one tab stop. This actually moves the whole line to the left one tab stop; Ctrl+T does the same thing to the right. Note that these keystrokes only work in Insert mode (use << and >> for the equivalent in Command mode).

tl;dr: set tabstop=4 softtabstop=-1 shiftwidth=0 expandtab

short form: set ts=4 sts=-1 sw=0 et

Explanation

If you set softtabstop (or sts) to -1 it will automatically behave the same as tabstop (ts), which will save you some hassle if you change tabbing a lot. Setting shiftwidth (sw) to 0 should effectively make that the same as tabstop as well.

In Detail

shiftwidth sw

Number of spaces to use for each step of (auto)indent. Used for cindent, >>, <<, etc.
When zero the tabstop value will be used.

tabstop ts

Number of spaces that a in the file counts for. Also see :retab command, and softtabstop option.

softtabstop sts

Number of spaces that a Tab counts for while performing editing operations, like inserting a Tab or using BS. It feels like Tabs are being inserted, while in fact a mix of spaces and s are used. This is useful to keep the tabs is setting at its standard value, while being able to edit like it is set to sts. When sts is negative, the value of shiftwidth is used. This will save you some hassle if you change tabstops a lot. When expandtab is not set, the number of spaces is minimized by using Tabs.

expandtab et

In Insert mode: Use the appropriate number of spaces to insert a . Spaces are used in indents with the > and < commands and when autoindent is on. To insert a real tab when expandtab is on, use Ctrl-V Tab. See also :retab