Do I really need an Anti-Virus software if I'm careful about where I browse?

As mentioned in some of the comments, there are no sites which can be guaranteed safe. Even reputable sites have suffered through banner ads, coding mistakes, deliberate attacks etc. so the first problem is that you cannot trust any website. You can work out a level of likelihood of safety by looking at the code from a sandbox and following links, but many attackers write code that hides from debugging tools or from testing environments, and code often changes attack vector with time.

So, to the next part - do you need an antivirus? Absolutely - a huge number of machines connected to the Internet are infected and running as part of botnets. If you don't protect your machine it may end up attacking mine. If you cannot detect malware on your machine I would blame your detection techniques, not think that your machine was clean.

From SANS, Expected infection time for an unprotected machine on the Internet is sub-5 minutes! (survivaltime is calculated as the average time between reports for an average target IP address ... aka someone pinging your computer would count as an "infection")

Sure, AV is only one layer of security (or potentially 2 if you use AV at gateway and desktop) but all layers have value. Any that you miss increases the likelihood of successful compromise.

There are many many angles to this question and I won't be able to cover all.

Up front a partial disclosure, I am working in the AV industry, but I won't give any more details. Another disclosure, I haven't been using any "realtime" AV mechanisms on my private machines in the past ten or so years. However, I have run offline scans on my machines from time to time.

Several angles to look at the problem

Drive-by-infections and trojans are but two of the potential attack vectors, so careful browsing will not completely relieve you of using common sense or doing your homework in other areas (keeping your software patched and such). There are many more, including targeted (usually known as "spear-fishing", "social engineering") attacks.

Underestimated potential for attacks

For example it has been shown by researchers that placing a USB key that looks like a flash drive but behaves like storage + mouse + keyboard can be used to get into otherwise highly secured areas. No one thinks of an attack first when finding a USB flash drive, right? But it could be.

Anyway, there are many more attack vectors on a wider scale. Most common examples include:

• vulnerabilities in software you use
  • Adobe Flash and Adobe Reader: PDFs (embedding JavaScript or Flash ... or nested combinations) - when on Windows, I suggest to use SumatraPDF, as it simply doesn't support scripts or Flash inside PDFs.
  • The various browser engines
  • Simple rendering of strings can exploit vulnerabilities (found in the past in font rendering)
  • Java when used as a browser plugin (although not really necessary any more these days unless you visit a lot of websites from the 1990s ;))
  • use of a number of complex and/or proprietary file formats (pictures, documents ...) where the software decoding the file format is more error-prone.
  • the operating system components themselves: think of the Blaster worm and WannaCry worm/ransomware.
• turning your own brain off at the wrong moment
  • getting an email from a loved one with an attachment? Uh oh, email senders can be faked like postcards. You can use cryptographically signed mails to guard against this. Better yet, you can use PGP/GPG to guard against anyone else but you and your loved one reading the contents by also encrypting it; the electronic counterpart of an envelope, just not as easy to "open".
  • actually going to some "erotic" site and just to see that one image or video executing that "downloader" or "viewer"
• Firmware can be "trojanized" so that your OS doesn't get the slightest chance to detect any wrongdoing on part of the computer.
  • Bluepill by Joanna Rutkowska was a similar approach in that it uses virtualization instructions to move the running system into the guest position while the rootkit itself retains the position as hypervisor.

To just name a few. Microsoft's own EMET has proven useful many times over in pre-empting certain kinds of exploits. But it won't be a silver bullet. If you are on a Windows edition that allows for it, you can also set Software Restriction Policies and Application Control Policies. Unfortunately Microsoft doesn't seem to give those a lot of love anymore and since the enforcement of SHA-2 hashed signatures, the ACPs don't work reliably anymore on the basis of the signatures. However, you can still use them with file hash rules.

Btw: the inverse is also true. One could also engineer a mouse or a keyboard that includes storage and triggers the execution of something off that storage. Here's a project that makes this accessible to a wider audience.

Wrong conclusions

But let's look at the quote you gave.

To be blunt: I refused to install any kind of antivirus or personal firewall software on most of my computers (but see Update 1/1/2012, below.) This included a Windows XP Home system that was used by my children as a web surfing / email / game system. I suffered zero infections during this time.

That clearly shows that the quoted person is not very knowledgeable about the subject. No one, including AV vendors can tell you with 100% certainty that you aren't infected. It's a silly claim. Just because there is no sign of infection doesn't mean there is no infection. A malware may lay dormant for a long time, for example, just to kick in at some particular point (think about the Michelangelo virus) There are very few methods, usually involving live CDs/DVDs (example Qubes) which will protect against most attack vectors, but one can usually come up with a scenario where they fail. The most practical are sandboxes that separate programs from each other and this is the approach that Qubes uses.

What I'm saying is that the person uses the wrong justifications and makes the wrong conclusions - not that there is anything wrong with not using AV programs.

AVs won't protect you necessarily, especially pure AVs

My question is, are there any documented studies that either prove or disprove these claims of being able to browse the internet carefully and having the same amount of risk of infection as that of browsing the internet with anti-virus software?

Again, browsing isn't the only attack vector through which malware can enter your system (the autorun mechanism in Windows together with a vulnerability in the icon handler for link files allowed Stuxnet to spread). It was as easy as plugging a flash drive into the computer. The next thing is that a pure AV solution will not filter the websites you access and will only trigger whenever something hits the disk (can also be the browser cache, though). So this makes it clear that a pure AV solution doesn't provide a very good protection against browsing-related infections in the first place.

However, we can assume that most malware needs to persist on the users' machines to be useful to its author - unlike in the past when malware authors proved their knowledge through the creation of viruses and such, these days malware writing is very much a quasi-commercial endeavor - such as skimming banking data or using the GPU for Bitcoin mining or ransoming and what not.

So when it hits the disk, the AV can catch it. Provided that the AV knows about it. And make no mistake, most AV vendors have long departed from the overcome signature-based detection or only use them as a supplementary detection method. That's contrary to popular belief that I have seen voiced time and time again. I am only aware of a single vendor that is basically sticking with the old ways of signature-based detection. The majority of vendors uses a detection that will catch usually hundreds or thousands of related malware samples at once. Still, the AV company has to have come across the malware in order to detect it. And the more samples it finds for a family, the better it can fine-tune its heuristics to avoid false positives.

It's a race and in the end also an arms race. Malware writers are using VirusTotal and other similar sites (as well as darknet versions of those services) in order to find out whether their creations get detected. Since there is money to be made, be it directly or by renting out your infected machine as a bot, there is enough incentive to keep a low profile.

Dispelling a myth ...

Just for those believers in AVs the results you see in many, if not most, of the certifications are severely skewed. That has to do with two things:

1. some of them want payment and that of course creates and obstacle for disgruntling the (paying) AV vendors
2. most of the tests allow for loopholes that are mercilessly abused in favor of test results to show off in the respective marketing reports of the vendors

In my opinion one of the better tests is the RAP (Reactive and Proactive) test which measures how good AVs are doing with "outdated" virus signatures. But there are certainly more things that would have to be improved in order to make these tests more objective.

Since I originally wrote this answer there's an effort underway, called AMTSO, to make tests more meaningful to anti-malware solutions' customers. At this time I am unsure it'll bring the promised improvements, but the future will tell. At the very least it has become clear that since the vendors outnumber the testers there seems to be a bias inherent in decisions and guidelines. But at least awareness of that issue exists.

Conclusion: you can't buy security

AVs will provide an additional layer of security to the layman and even to the expert. Period. But - and that's a big but - this added security can never offset lack of education in security-aware computer use. Conversely I hold that, yes, if you are paranoid and diligent enough you don't need an AV as in a must have. The technological layer of protection will not offset the protection you gain from due diligence, paranoia and most importantly awareness.

The security provided by an AV can very much be compared to airport security. There's a lot of security show, but there's also some actual added security involved. But don't believe that the AV will protect you - or others for that matter - a 100% from getting infected. Which can also be read as a shameless plug for any backup solution (be sure to keep backups offline, so they don't get wiped by ransomware, for example).

In my opinion the best protection by far is awareness and diligence. But unfortunately there is nothing like a driver's license required for computer or internet use. So failing that the next best protection would be sandboxes, whitelisting approaches and also AVs and other anti-malware solutions in addition to regular offline-stored backups. They'll never be able to give you a 100% protection (well, whitelisting might), but they can protect you from a wide range of prolific threats. So when in doubt, sure go for that AV (or in general anti-malware) solution. But that's where it gets difficult. They differ in quality and even the tests - as I pointed out above - are often severely skewed. Adding another AV engine into the mix may help but there's no guarantee. If inside of a single product, the integration of several engines will likely be good and not add too much overhead. However, if you install several AV products in parallel you are asking for trouble. In fact modern Windows versions only allow you to see the stats of a single security solution per category in the Windows Action/Security Center.

As for whitelisting, recent Windows versions are pretty cool about that. You can effectively implement this yourself (run gpedit.msc) in the Software Restriction Policies and the Application Control Policies (or your admin can do that if you're running in some managed environment). I think it was introduced at least since Windows 7, but it's possible that these facilities even existed on Windows Vista. Unfortunately this feature may depend on the edition of your Windows version, which is a shame. There's also a catch in that if you base the rules on code-signatures, this may fail with executables that are only SHA-2 signed.

Also:

keep in mind: Strictly speaking we can never tell whether a system is clean, whereas we can state that we didn't find anything. Every claim to the contrary is either Marketingese or is going to obsolete the AV industry as a whole quickly.

(source: my answer over on SuperUser here)

A yet better security net are backups and frequent rollbacks to a known-clean state - but that's just in my opinion.

Commercialized malware doesn't need privileged access, not even on Linux, FreeBSD or macOS

Over on SuperUser I wrote this:

One more thing. Many Linux (also MacOSX) users fall prey to the assumption that malware that doesn't have privileged access can't hurt you (which is also often cited as a reason why there "isn't" malware for unixoid systems, which also isn't entirely true). This couldn't be further from the truth. Although this prevents it from establishing a system-wide stronghold, it won't keep malware from skimming data from your personal files etc. A rogue browser extension installed in your own profile will still be as dangerous to your account as the one that can do it to all accounts. If you do your internet banking with the rogue extension installed, it makes no difference whether you are root or joe.

I'm going to make an analogy: Do I need a condom when I'm really careful when having sex? Yes you do because even when you are safe you don't know the other party involved is even if he or she says they are.So do you need to use a condom/anti virus? Of course you do, better be safe than sorry as getting rid of a virus is very hard.

Do note that even when using an anti virus things can go wrong even when you are being safe! If you suspect things have gone wrong it is best to restore from backup or perform a clean install.