Is HTML5 vibrate feature a security vulnerability?

A popup was used to show the alert. Does this mean that the popup feature introduces vulnerabilities? Then by that line of reasoning JavaScript is the source of all problems. There are people who actually think that JS is an important vector for attacks and block it on untrusted websites with extensions like NoScript.

Many features can be misused, and is up to people creating the standards, browsers and even websites to judge what is bad and to change the standards or implement mitigations. Of course those people can be wrong and some feature can be unexpectedly used to attack users.

A nice example is the browser's console which is very often used to trick users into pasting JS code that attacks the user. This helped Facebook worms to propagate with great success. Facebook noticed this and introduced this message in the console:

This vibrate function might trick some users into thinking that it is actually the OS showing the alert, but I think the latest mobile browsers do a good job of showing the user that he is still inside the browser. In this case, the message from the browser is clear enough "The page at andro-apps.com says:"

If this becomes an important vector for attack, I'm sure the browser manufacturers will notice that and will make changes to reduce the impact.

Suppose a malicious web page pops up a fake system notification and vibrates at the same time. How confident would you be of telling the difference between a legitimate pop-up and a .png on the web page you're viewing.

(Source)

Personally I have not heard of any exploit related to HTML5 Vibrate API, but it could be used for evil goals as shown on the link above. But more serious is what the quoted text above mentions: you can not distinguish between a legitimate pop-up and something else. This something else could be a pop-up used to trigger a drive-by download attack leading to malware (usually spyware or adware) installation on your system by exploiting the vulnerabilities of the browser you use (or those of its plugins).

But luckily, I was able to calm my nerves sufficiently to realize that this is a scare-ware served through an ad-server and that the anti-virus could be the actual virus.

You have rather been wise in your decision because it could be a drive-by download attack. Try to use free (but powerful) services such as Stop Badware on your laptop to see if the website you surfed is blacklisted (notification may be negative in case the website is compromised too recently and no one reported it).

Honestly, the core question is whether vibration of the phone will give an application/website significantly more authority than without the vibration. Now, obviously I lack any research into this specific issue, but we can note that applications do not use vibrations as a way to convene authority. If anything it would feel wrong for an application to vibrate whilst turned on and be an extra indicator that something is odd about the situation, as vibration tends to only be triggered when the screen is off.

Might vibration create a feeling of urgency for some people? Definitely, and thus it might marginally improve the efficiency of scareware, but even if that would be the case it would still not be a security vulnerability as the vibration API would not allow one to do anything that one isn't allowed to do which is a necessary trait of a security vulnerability. So in conclusion it's definitely not a security vulnerability and it would make little sense to lock it away behind a permission dialog.