Why doesn't a braking car move backwards?
A notable property of frictional forces is that they resist motion (as opposed to other types of forces, which might resist displacement, for example, which is how a spring behaves). As a result, the brakes on your car slow down the motion of your wheels that produces forward movement of your car—but they also slow down motion that produces reverse movement.
If instead you used another type of force-applying system to slow down your car (e.g., a giant spring), then your car would slow down, then stop, and then start moving backwards.
Cars move because the wheels are spinning in a certain direction. Brakes work by making the wheels not spin, not by making them spin in the opposite direction.
If instead of slamming the brakes you "brake" a car by having some other kind of force pushing it backwards, like a super huge fan in front of it, then yes, it might begin moving backwards.
Friction as loss of energy rather than force
Two objects moving relative to each other have a relative kinetic energy proportional to the speed at which they are moving and the rotation they suffer.
When the surfaces of these objects are pressed together, friction occurs. This friction is just a process through which this relative kinetic energy is transformed into thermal energy, causing a temperature rise in both objects, and "sound energy" (the metallic screech that can be heard sometimes when braking), but this last part is negligible.
When applying this to the wheels of a car and its brakes, we see that the brakes and the wheels have no relative speed but, while the wheels rotate, the brakes do not move, as they are fixed to the car. This rotation only occurs when the car is advancing.
When you press the brake pedal, the system in your car presses the brakes against the braking surface in the wheels, creating friction and dissipating the kinetic energy of the wheels, which in turn slows down the car. As @senderle points out in his comment, some of the energy dissipated can be reused in cars, e.g. recharging the batteries. This happens until there is no more relative motion between the wheels and the brakes.
This means one of two things:
The car has stopped and the wheels are at rest. No more relative motion so no more energy to dissipate. The car cannot go backwards because the car has no energy (movement means kinetic energy, and the car has none of it), and to make it move you have to provide it with energy, not take it away (in classical mechanics you cannot have negative kinetic energy, so 0 is the lowest energy state)
The car is skidding (see @Eric Lippert's comment). This means that energy is dissipating only in the friction between the rubber of the wheels and the ground. The rate of dissipation in this case is incredibly lower that that of normal braking. This is the reason ABS is implemented in modern vehicles.
Note: In my country, when the wheels stop rotating but are sliding over the ground is called "wheel locking". I don't know how it is called in english, so if anyone knows please let me know so I can correct it if needed.