What is V.A.T.S in Fallout 3?

It's the system in which you can pause the action a take aim at specific body parts, with known percentage of hitting them, and with each action costing a specific amount of "action points" (AP).

It stands for "Vault-Tec Assisted Targeting system", but mostly it

• Gives a more cinematic feel when using it, due to the slow-motion action sequence after choosing a target.
• Makes the game closer to earlier, turn-based Fallout games in which you could target specific body parts with known percentages, paying with action points, in a very similar way.
• Gives the game a more RPG-ish, tactical feel than a typical shooter.
• Makes the game easier / less stressful.

You can read more about it on Wikia's Fallout, Gamespy and even Wikipedia.

It's the Vault-Tec Assisted Targeting System

See here

The Vault-Tec Assisted Targeting System, or V.A.T.S., is a queuing system for Fallout 3 and Fallout: New Vegas, inspired partly by the ability to shoot specific body parts in the turn-based combat system of Fallout and Fallout 2. V.A.T.S. is also used for a cinematic combat experience.