Should anti-virus detect metasploit payloads?

It's an arms race. The developers of metasploit want to develop plugins that defeat anti-virus. The developers of anti-virus want to defeat metasploit plugins.

They can't both be successful, so sometimes the AV will roll out signatures that detect all metasploit modules, sometimes the metasploit developers will find a new way to evade AV.

You'd think that the AV vendors had the advantage due to metasploit being open-source, but obviously not in this case.

The logical answer is that yes, since Metasploit is largely open source, all AV should detect and block Metasploit generated modules if they are doing their jobs. Unfortunately, the reality is that it is incredibly hard to actually detect and block malicious code/executables even if it is generated with a open source framework such as Metasploit. My take on the matter is simply that Metasploit had a new update and AV vendors have not created signatures for the new generated payloads yet.

Here are some of the reasons:

1. It is an arms race as TimC mentioned. AV companies find ways to block malicious code via signatures and heuristic detection. This in turns lowers the effectiveness of products and tools designed to bypass AV so new ways of bypassing AV is found and incorporated in the avoidance products. The developers and community contributors of Metasploit understand AV products exceedingly well and with the constant updates, there are often new innovative techniques that will for a period of time avoid AV until the vendors catch up again.

2. The defenders do not share information effectively and this results in what is detected on one AV might not get flagged by another for quite some time, or ever. While work is progressing towards a standard (http://standards.ieee.org/develop/indconn/icsg/malware.html), business is business and a lot of vendors have their own proprietary standards effectively hindering information sharing. A test by LastLine labs showed just how long it can take for malware detection to be achieved by vendors. PLEASE NOTE: They used, Virustotal for the test and this is NOT a real world test but it does provide interesting information. For more information regarding using VirusTotal for testing, see this: https://community.webroot.com/t5/Techie/Testing-antivirus-with-VirusTotal-is-a-bad-idea/td-p/62881 . It basically boils down to not having the ability to use all detection mechanisms in the VirusTotal environment.

However, I wondered if this behaviour might in fact be intended? Are there legitimate arguments that "metasploit is not a virus"?

No AV vendor would willingly white list Metasploit and I am sure they would gladly classify it as a virus if they could. Proof of this is quite simple, try to install Metasploit on a machine with AV running!