Roundup of Antivirus Tests 2011

Discussion in 'other anti-virus software' started by SM_Unlimited, Jan 19, 2012.

Thread Status:
Not open for further replies.
  1. SM_Unlimited

    SM_Unlimited Registered Member

    Joined:
    Jun 8, 2010
    Posts:
    31
    Hey all. I have created a roundup of all the testing results throughout the year (again :p) to find the overall detection capabilities of antivirus products. Considering others may find these statistics useful in general i have decided to share. Trusty excel skills were in use to find out just for my own purposes to see how my own current choice in antivirus software is holding up.

    Current testing results used from the year:
    AV-Test Quarterly Tests: http://www.av-test.org/certifications.php
    AV-Comparatives: http://www.av-comparatives.org
    VBulletin (VB100 only): http://www.virusbtn.com
    http://www.chip.in/images/content_images/june_2011/Security-Suites-comparison.jpg.jpg
    Passmark:http://www.passmark.com/ftp/antivirustesting-ed2.pdf
    http://www.passmark.com/ftp/ESet_vs_9_Competitors_InternetSecuritySuites-Sept2011.pdf
    Malware Research group (PUA and flash Tests): http://malwareresearchgroup.com/malware-tests/
    Antimalware.ru: http://translate.google.es/translat...re.ru/antivirus_self_protection_test_x64_2011
    http://www.anti-malware.ru/malware_treatment_test_2011
    Dennis Labs: http://www.dennistechnologylabs.com/reports/s/a-m/symantec/DTL_PCTP2011_SYMC.pdf http://www.dennistechnologylabs.com/reports/s/a-m/symantec/DTL_PCTP2012_SYMC.pdf
    http://www.cert.hr/sites/default/files/NCERT-LAB-PUBDOC-2011-02-001.pdf
    http://www.maximumpc.com/article/features/ultimate_antivirus_guide_10_top_programs_reviewed?page=0,2
    http://www.maximumpc.com/article/fe..._10_anti-virus_programs_reviewed_and_compared

    Summary Information:
    Uncertainty lists the % of samples out of the total that have been estimated due to lack of results. The result will be normalised based on the results of the other products in a test set so as long as total uncertainty is low it is a reliable score, just not 100% guaranteed to have that exact score. I have collated results from a wider number of products this year, usually less tested hence uncertainty results are higher this year. A product is not listed if it has not been tested.

    The results are categorised into: On Demand, Retrospective, Dynamic Real World, Removal, Performance,False positives and the King of Tests

    On Demand: Raw detection capabilities of signatures when fully updated with latest definitions.
    Retrospective: Capabilities of heuristics and generic signatures against new unknown malware.
    Dynamic: Real world protection capabilities using all components of the security software to demonstrate effectiveness in the real world (0 day, cloud, behaviour blocking, self defence, web blocking, etc)
    Removal: Effectiveness at cleaning a system from malware, the higher the score the better the product is at disinfection and removal of malware trace elements without damaging the system in the process.
    Performance score (higher is better) :How fast and responsive products perform. The higher the result the less of an impact on your system.
    only helps to provide comparison of performance between products.
    False Positives: Total false positives across all tests with such information available, false warnings count as 0.5. The results help to show which products are more prone to false positives.
    King of Tests: Personalised scoring system in an attempt to determine which antivirus solutions did the best across all the categories equally (This year is also weighted on uncertainty). Products with >50% mean uncertainty are not included.

    AV2011-1.png
    AV2011-2.png
     
    Last edited: Jan 20, 2012
  2. Sher

    Sher Registered Member

    Joined:
    Oct 19, 2005
    Posts:
    366
    Location:
    Pakistan
    Awesome work!!!
     
  3. HealingStargate

    HealingStargate Registered Member

    Joined:
    Jan 3, 2009
    Posts:
    160
    Location:
    USA
    Very good and helpful. Great job well done. Thank you.
    KOR-
     
  4. Motherroad

    Motherroad Registered Member

    Joined:
    Feb 13, 2006
    Posts:
    234
    Location:
    Florida
    Very nice. Thanks.
     
  5. Brandonn2010

    Brandonn2010 Registered Member

    Joined:
    Jan 10, 2011
    Posts:
    1,849
    Well this is phenomenal. Some tests results are much different than another test but this big picture really helps. I noticed that 4 out of the top 5 King of Tests were AV-Comparative's winner and Top Rated.

    Also, what is the Uncertaintly %?
     
  6. Frank the Perv

    Frank the Perv Banned

    Joined:
    Dec 16, 2005
    Posts:
    882
    Location:
    Virginia, USA
    Just Really Awesome work.

    Cheers.


    :thumb: :thumb: :thumb:
     
  7. RejZoR

    RejZoR Registered Member

    Joined:
    May 31, 2004
    Posts:
    6,426
    Very useful, though i just don't understand the "Uncertainty" entry. What does it mean exactly?
     
  8. Atul88

    Atul88 Registered Member

    Joined:
    Dec 8, 2011
    Posts:
    259
    Location:
    India
    Performance :
    AVG better than Avast?:shifty:
    Norton lighter than Webroot:blink:
    that means i can keep ESET & norton both together?
    Hurray!!!
    BTW That's some professional stuff..
    Good Job, Keep it Up
     
    Last edited: Jan 20, 2012
  9. Romagnolo1973

    Romagnolo1973 Registered Member

    Joined:
    Feb 17, 2009
    Posts:
    518
    Location:
    Italy - Ravenna
    thanks !!! Great work
     
  10. Nevis

    Nevis Registered Member

    Joined:
    Aug 28, 2010
    Posts:
    786
    Location:
    255.255.255.255
    Good Work :thumb:
    I am also dubious about Uncertainty
     
  11. Narxis

    Narxis Registered Member

    Joined:
    Jun 10, 2009
    Posts:
    477
    Great work!
     
  12. ALiasEX

    ALiasEX Registered Member

    Joined:
    Mar 30, 2010
    Posts:
    240
    Most of the Webroot tests are not of Webroot SecureAnywhere. (I think this could be the cause of your surprise)
     
  13. Atul88

    Atul88 Registered Member

    Joined:
    Dec 8, 2011
    Posts:
    259
    Location:
    India
    You mean to say it was with the previous version!!!
     
  14. Romagnolo1973

    Romagnolo1973 Registered Member

    Joined:
    Feb 17, 2009
    Posts:
    518
    Location:
    Italy - Ravenna
    can you explain better what is the Uncertaintly % ? is not so clear, thanks
     
  15. SM_Unlimited

    SM_Unlimited Registered Member

    Joined:
    Jun 8, 2010
    Posts:
    31
    For example taking Microsofts on Demand result.

    It had an uncertainty of 10.65%. That means 10.65% of 2544331 samples are estimated as microsoft was not tested with those samples. For such estimates it instead takes the mean average result for the tests it doesn't take part in.

    Products not listed are akin to an uncertainty of 100%
     
  16. trjam

    trjam Registered Member

    Joined:
    Aug 18, 2006
    Posts:
    9,057
    Location:
    North Carolina
    so the 91 percent for Vipre means that Vipre really wasnt tested with 91 percent of the malware. Doesnt make sense.
     
  17. pbust

    pbust AV Expert

    Joined:
    Apr 29, 2009
    Posts:
    1,173
    Location:
    Spain
    Very interesting, thanks!

    Is it possible to separate PandaCloudAV from the other Panda products (AVPro and Internet Security)? All Passmark, antimalware.ru and Dennis tests are older Panda products while AVT, MRG and AVC are with CloudAV. Otherwise the global score is really impacted negatively by the older products and not very accurate, but I guess it would also raise the uncertainty for each.
     
  18. SM_Unlimited

    SM_Unlimited Registered Member

    Joined:
    Jun 8, 2010
    Posts:
    31
    In 2010 it swung both ways, but I agree. The main reason I havn't done so is it greatly increases the amount of effort to create these results (Every AV-company has at least 2-3 different versions of AV).

    As you noticed this would also dilute the limited testing results available even more. In a perfect world every companies product portfolio would be tested in every possible test but that doesn't happen :p.
     
  19. Securon

    Securon Registered Member

    Joined:
    Jan 11, 2009
    Posts:
    1,935
    Location:
    London On
    Good Evening! Quite obviously a labour of Love. Many Kudos. Sincerely...Securon
     
  20. toxinon12345

    toxinon12345 Registered Member

    Joined:
    Sep 8, 2010
    Posts:
    1,200
    Location:
    Managua, Nicaragua
    What does the decimal for the total false positives? If expressed using percentages, or average what is the meaning?
     
  21. SM_Unlimited

    SM_Unlimited Registered Member

    Joined:
    Jun 8, 2010
    Posts:
    31
    The decimal points come from the estimated results as that causes them to no longer be an exact number of false positives. There is a slight exception as any false warnings also only count as 0.5 each rather than 1 each.
     
  22. Dark Shadow

    Dark Shadow Registered Member

    Joined:
    Oct 11, 2007
    Posts:
    4,553
    Location:
    USA
    No doubt a lot of time and effort went in to this,Excellent work.thanks
     
  23. Q Section

    Q Section Registered Member

    Joined:
    Feb 5, 2003
    Posts:
    771
    Location:
    Headquarters - London & Field Offices -Worldwide
    We wish to again bring up a point for everyone's consideration.

    Scenario one would be that all malware in a system is found and reported (and dealt with).

    Scenario two is that all malware that attempts to execute on a system is dealt with before it does any harm paying less care to any malware that may be present but not executing (or causing any changes) hence at that time is no threat to the system until if and when it executes and is not caught by a security program.

    So to say such and so a security program (antivirus) "found" more malware (perhaps idle and causing no harm) than another is really not the best consideration. Which program stops the most malware from executing is most important we believe.

    So the different philosophies of methods of defense by the various antivirus companies usually are not taken into consideration but should be.

    Many may say "But I do not want any malware on my computer". We say "We do not wish any malware to execute on any of our computers".
     
    Last edited: Jan 21, 2012
  24. Yash Khan

    Yash Khan Registered Member

    Joined:
    Apr 4, 2011
    Posts:
    1,837
    Amazing work done!

    Thanxx for your time & efforts.

    Regards
    Naren
     
  25. xandros

    xandros Registered Member

    Joined:
    Oct 30, 2006
    Posts:
    411
    thank you
     
Loading...
Thread Status:
Not open for further replies.