AV-Test.Org Releases Real-World Malware Protection Report

Article:

http://blogs.pcmag.com/securitywatch/2009/12/av-testorg_releases_real-world.php

 

PC Tools is the real stunner here ... Guess the infusion of Threatfire in PCT IS 2010 has yielded rewards.
Bitdefender is the biggest disappointer. Pushed to the bottom rug !!

Rest of the result is on expected lines. A little disappointed by McAfee. I expected their Cloud-Detection to be better.

This is probably the best test I have seen. Real world scenario with large sample set distributed over timeline. Kudos to AV-Test.org for this labour intensive test.

 

Man PCTools is doing good these days in the tests. Their firewall also scored good in the Firewall test (matousec)

 

Norton and their free security products, PCToold are the winners
Which was the AV engine of pctools?

 

PC Tools licenced VirusBuster engine and has modified it for their use and DB.

 

I hope this is the beginning of a trend toward more real-life testing of security suites.

 

This test is very impressive: a 60-day evaluation of real-world performance.

Two observations...

• The alignment of the findings of this test to the similar one funded by Symantec and conducted by Dennis Technology Lab further weakens the argument that Symantec “influenced” the outcome of that test.
• What is most surprising in this test, in my opinion, is the magnitude of the performance gap between the top performers (e.g., Norton, Kaspersky, PC Tools) as compared to the remainder (especially Avira, McAfee, CA, F-Secure, BitDefender and Trend Micro).

 

http://www.darkreading.com/security/antivirus/showArticle.jhtml?articleID=222002625

Looks like the knifes are out !!!

 

Readers of this thread may be intersted in further published commentary on this test...

However, NSS Labs has its own perspective to add...

The comments by NSS Labs (even though they are a well-respected organization) may be dated, since it does not appear that they have tested the 2010 editions of the anti-malware products (see here).

 

It's good to see this kind of a test. It's long over due and nice to see the shift toward this the numbers are kind of what I expected. It shows who has been improving the product not just the detections I hope to see more of these in the future.

 

Something fishy there. Norton/Symantec/PcTools topping.

I trust AV-Comparatives.org more.

 

I don,t know about the details but the very good result of PC tools may be due to the HIPS component of their fire wall.

 

People have such a hard time believing Norton being on top. I don't see why its so hard to believe they have more Market share, Manpower, Resources and Time in this business. Who knows maybe it's just biased is what blinds people.

 

Norton/Symantec is ok. Just tell more about PC Tool's engine.

I have tried them all...almost.

 

I think that is whats important to note, and something that I wonder if Rick Moy's comments took into account, or if in fact his comments are dated (as Pleonasm suggests) or taken out of context. However he is later quoted as saying "Focusing on just 10 zero-days does not reflect the current threat spectrum on the Internet."
Which seems to suggest he is talking about the current AV-Test, but that he might be mistaken as I believe the total test set is 600.

As quoted from the original link...
"By contrast, tracking the 600 malware samples and 400 clean files took all of AV-Test's resources."

"The current test simply evaluated whether the product detected each malware sample, without regard for which security component handled the detection."

Considering all the resources used to conduct this test, I think it would have been some what interesting if a more detailed report was made available (perhaps separately) that included what security component of the AV Suite blocked the malware attack. Or at least a simple distinction between whether the antivirus component blocked the attack or other component of the suite.

 

I find the whole testing business highly entertaining. After having us bombarded for years with the "on-demand" tests, now they say to forget all about what we know , because now they do REAL WORLD tests. Which, by coincidence, have brought upside down the world of "antivirus detection" as we knew it.


So basically, everyone that was saying that CA is crap, has now to think again, because it beats Bit Defender and F-Secure which were "top dogs" before. And Avira should hand the crown to... PC Tools!

I love AV tests! They bring life to a forum! Within 5 months, everyone will have forgotten of the "old kings" and will be accepting the new natural order of things.

- Before: "I have the best of the best: Avira Free! AVG? Does it even detect anything?"
- After 5 months: "Dude, don't get Avira! It's decent at best, but that's all. Get PC Tools or AVG Free, the best freebies"!

Until some other "more real life" test method comes out which will shatter our realities once again.

I 'd say pick an AV you like, do some of your own tests and be happy. AV tests are a relative strength indicator for me and nothing more.

 

The main thing I do like about this test. It tests the product as a whole not just one portion of it. Makes it so you can see what your product really does protect against.

 

I agree... I like that it tests all aspects of a product.

What I find unusual is that with 10 fresh threats introduced each day for 60 days, it sounds as though the systems are not cleaned after each day, so an infected system (or a completely healthy one, depending) is essentially attacked repeatedly. If a threat goes undetected, it seems like the compromised machine would be crippled to some extent, and even more prone to further infection. I don't know if clinically that is correct, but it sure does emulate real world conditions.

 

Quick graph making it easier to see what the AV-Test statistics mentioned in the initial post is showing:



What is the difference between the "detection" and "blocking"? Shouldn't detection = block?

 

Free versions of Prevx detects, but does not block (most threats). Paid version of Prevx protects, or blocks.

 

That seems as likely an explanation as any....

Bias is defined as “a particular tendency or inclination, esp. one that prevents unprejudiced consideration of a question” (source: dictionary.com).

Yes, looking at the protection provided by the anti-malware product in total under real-world conditions logically seems to be the most accurate way to assess performance.

In my opinion, this test by AV-Test is now the "gold standard" against which all other anti-malware comparatives must be judged.

An interesting question....

Nonetheless, if you look at the paper by AV-Test entitled Testing of “Dynamic Detection” (available here), the ideal test scenario is described as: “Record the actions of the security software, and compare this record to actions of the clean base system.” The phrase “clean base system” in this paper seems to suggest that AV-Test would have restored a backup image to the PC following the introduction of each tested malware threat.

The same paper also states "The clean state of the operating system has to be known" as a prerequiste for testing.

 

This is for Suites only. Last I heard Comodo was staying out of tests until they felt there AV was on par.


Edit.
The person I was quoting has ether deleted his post or been removed by a mod. So if it seems out of place that is why.

 

Interesting test, interesting results

I find it surprising that PC Tools did well. Very glad Panda did well

 

But that doesn't mean that this test was done that way, does it? All this current test says is... "Each day for 60 days, researchers released 10 fresh threats on the test systems and analyzed each product's ability to detect the threat and to fully block its installation." To me, that implies that the systems were left intact. I'd like to know for sure.

 

Note that the reported “detection rates” also include “warning messages.” If a user is warned that a download is potentially malicious and still executes the file, the fact that the anti-malware product doesn’t block the threat (while undesirable) should not detract, in my opinion, from the assessment of the quality of the product. No product can protect against all “foolish decisions” by users, after all.