How to detect hosts running in virtual machines with nmap?

You could use the Metasploit Framework HTTP Virtual Host Brute Force Scanner module.

Nmap is best launched from inside Metasploit. See the Metasploit Unleashed (free training available from Offensive-Security) section on Port Scanning for more information.

If the target IP address is available from the global Internet, then I suggest you also check out MyIPNeighbors and SHODAN, which are incredibly resourceful for this sort of reconnaissance activity.

Identification of caching servers, CDN infrastructure, reverse web proxies, load-balancers, internal IP prefixes, archived content, and associated hosts may also help during the investigation of virtual hosted infrastructure. Be sure to check out tools such as the Host/IP Pattern Extraction Tool (host-extract.rb), Halberd, The Web Archive, HTTP Archive, and W3AF.

To circle back to your original question, it is possible to scan for vhosts using an NSE (Nmap Scripting Engine) script called http-vhosts. However, it's good to know the intricacies of the entire target architecture before settling on a final decision about what has been / has not been discovered.

Typically, I find it easier to first find a path disclosure vulnerability that leads to a file read inclusion vulnerability -- and then to download the web server configuration in order to go through it manually. Or email/phone/text/DM/FaceBook-message the web server administrator who has access and then ask him or her for a copy of the web server configs.

If you are on local LAN (force with --send-eth), you can map the MAC Address' OUI 12-bit extension identifier map to the org assigned by the IEEE Registry Authority. Nessus and some other tools do this by default. My favorite is NetworkMiner of the traffic, as it also provides a sample of packet analysis and gives a good view of what happened for proactive troubleshooting.

The following link has a good overview of this :

http://lab.lonerunners.net/blog/virtual-host-and-dns-names-enumeration-techniques

Covers: 1. Why you need to enumerate 2. Techniques 2.1 DNS enumeration techniques 2.2 Banner grabbing 2.3 SSL/TLS Protocol enumeration techniques 2.4 HTTP Protocol enumeration techniques 2.5 Passive web enumeration techniques 2.6 Active web enumeration techniques

If it is a local subnet, you can get the MAC addresses of the hosts and check if they belong to vmware or other companies. Usually nobody bothers changing them. nmap will tell you too with -A switch etc.