Are there downsides to using V.A.T.S?

Aside from AP limitations, there is also the fact that VATS drains Fusion Cores more quickly, as mentioned by @HopelessN00b. Also keep in mind that aiming manually may have tactical advantages over VATS depending on circumstances.

Those circumstances include:

• Enemies that are behind heavy cover may have a 95% chance of getting hit in VATS, but your VATS shot will always hit the cover instead.
• Enemies may hide behind cover while you're still shooting using VATS, in which case you'll have to cancel VATS to stop wasting your AP.
• VATS loses a lot of accuracy over long distances, even if you have a sniper rifle equipped and your enemy is hardly moving.
• VATS does not allow you to move. If you need to dodge while attacking, VATS is not for you.
• You are vulnerable while targeting. VATS does not stop time, it only slows it. While you're deciding which enemy or body part to hit, you remain cannon fodder for others. Fortunately, as you receive 90% damage resistance while in VATS, this point is not as bad as it might seem.
• VATS is generally a bad fit with heavy weaponry. Rockets/Nukes may be launched even when the enemy has gotten too close. Miniguns will be shot in salvos which will either all hit a single target, or all miss (very bad if you're being overrun and don't care about hitting a single target).

In addition to the the points brought up in Nolanar's answer, VATS burns through fusion cells very quickly, which is a serious drawback for power armor users.