Roundup of Antivirus Tests 2011

Hey all. I have created a roundup of all the testing results throughout the year (again ) to find the overall detection capabilities of antivirus products. Considering others may find these statistics useful in general i have decided to share. Trusty excel skills were in use to find out just for my own purposes to see how my own current choice in antivirus software is holding up.

Current testing results used from the year:
AV-Test Quarterly Tests: http://www.av-test.org/certifications.php
AV-Comparatives: http://www.av-comparatives.org
VBulletin (VB100 only): http://www.virusbtn.com
http://www.chip.in/images/content_images/june_2011/Security-Suites-comparison.jpg.jpg
Passmark:http://www.passmark.com/ftp/antivirustesting-ed2.pdf
http://www.passmark.com/ftp/ESet_vs_9_Competitors_InternetSecuritySuites-Sept2011.pdf
Malware Research group (PUA and flash Tests): http://malwareresearchgroup.com/malware-tests/
Antimalware.ru: http://translate.google.es/translat...re.ru/antivirus_self_protection_test_x64_2011
http://www.anti-malware.ru/malware_treatment_test_2011
Dennis Labs: http://www.dennistechnologylabs.com/reports/s/a-m/symantec/DTL_PCTP2011_SYMC.pdf http://www.dennistechnologylabs.com/reports/s/a-m/symantec/DTL_PCTP2012_SYMC.pdf
http://www.cert.hr/sites/default/files/NCERT-LAB-PUBDOC-2011-02-001.pdf
http://www.maximumpc.com/article/features/ultimate_antivirus_guide_10_top_programs_reviewed?page=0,2
http://www.maximumpc.com/article/fe..._10_anti-virus_programs_reviewed_and_compared

Summary Information:
Uncertainty lists the % of samples out of the total that have been estimated due to lack of results. The result will be normalised based on the results of the other products in a test set so as long as total uncertainty is low it is a reliable score, just not 100% guaranteed to have that exact score. I have collated results from a wider number of products this year, usually less tested hence uncertainty results are higher this year. A product is not listed if it has not been tested.

The results are categorised into: On Demand, Retrospective, Dynamic Real World, Removal, Performance,False positives and the King of Tests

On Demand: Raw detection capabilities of signatures when fully updated with latest definitions.
Retrospective: Capabilities of heuristics and generic signatures against new unknown malware.
Dynamic: Real world protection capabilities using all components of the security software to demonstrate effectiveness in the real world (0 day, cloud, behaviour blocking, self defence, web blocking, etc)
Removal: Effectiveness at cleaning a system from malware, the higher the score the better the product is at disinfection and removal of malware trace elements without damaging the system in the process.
Performance score (higher is better) :How fast and responsive products perform. The higher the result the less of an impact on your system.
only helps to provide comparison of performance between products.
False Positives: Total false positives across all tests with such information available, false warnings count as 0.5. The results help to show which products are more prone to false positives.
King of Tests: Personalised scoring system in an attempt to determine which antivirus solutions did the best across all the categories equally (This year is also weighted on uncertainty). Products with >50% mean uncertainty are not included.

 

Awesome work!!!

 

Very good and helpful. Great job well done. Thank you.
KOR-

 

Very nice. Thanks.

 

Well this is phenomenal. Some tests results are much different than another test but this big picture really helps. I noticed that 4 out of the top 5 King of Tests were AV-Comparative's winner and Top Rated.

Also, what is the Uncertaintly %?

 

Just Really Awesome work.

Cheers.

 

Very useful, though i just don't understand the "Uncertainty" entry. What does it mean exactly?

 

Performance :
AVG better than Avast?
Norton lighter than Webroot
that means i can keep ESET & norton both together?
Hurray!!!
BTW That's some professional stuff..
Good Job, Keep it Up

 

thanks !!! Great work

 

Good Work
I am also dubious about Uncertainty

 

Great work!

 

Most of the Webroot tests are not of Webroot SecureAnywhere. (I think this could be the cause of your surprise)

 

You mean to say it was with the previous version!!!

 

can you explain better what is the Uncertaintly % ? is not so clear, thanks

 

For example taking Microsofts on Demand result.

It had an uncertainty of 10.65%. That means 10.65% of 2544331 samples are estimated as microsoft was not tested with those samples. For such estimates it instead takes the mean average result for the tests it doesn't take part in.

Products not listed are akin to an uncertainty of 100%

 

so the 91 percent for Vipre means that Vipre really wasnt tested with 91 percent of the malware. Doesnt make sense.

 

Very interesting, thanks!

Is it possible to separate PandaCloudAV from the other Panda products (AVPro and Internet Security)? All Passmark, antimalware.ru and Dennis tests are older Panda products while AVT, MRG and AVC are with CloudAV. Otherwise the global score is really impacted negatively by the older products and not very accurate, but I guess it would also raise the uncertainty for each.

 

In 2010 it swung both ways, but I agree. The main reason I havn't done so is it greatly increases the amount of effort to create these results (Every AV-company has at least 2-3 different versions of AV).

As you noticed this would also dilute the limited testing results available even more. In a perfect world every companies product portfolio would be tested in every possible test but that doesn't happen .

 

Good Evening! Quite obviously a labour of Love. Many Kudos. Sincerely...Securon

 

What does the decimal for the total false positives? If expressed using percentages, or average what is the meaning?

 

The decimal points come from the estimated results as that causes them to no longer be an exact number of false positives. There is a slight exception as any false warnings also only count as 0.5 each rather than 1 each.

 

No doubt a lot of time and effort went in to this,Excellent work.thanks

 

We wish to again bring up a point for everyone's consideration.

Scenario one would be that all malware in a system is found and reported (and dealt with).

Scenario two is that all malware that attempts to execute on a system is dealt with before it does any harm paying less care to any malware that may be present but not executing (or causing any changes) hence at that time is no threat to the system until if and when it executes and is not caught by a security program.

So to say such and so a security program (antivirus) "found" more malware (perhaps idle and causing no harm) than another is really not the best consideration. Which program stops the most malware from executing is most important we believe.

So the different philosophies of methods of defense by the various antivirus companies usually are not taken into consideration but should be.

Many may say "But I do not want any malware on my computer". We say "We do not wish any malware to execute on any of our computers".

 

Amazing work done!

Thanxx for your time & efforts.

Regards
Naren

 

thank you