AV-Test.Org Releases Real-World Malware Protection Report

Discussion in 'other anti-virus software' started by King Grub, Dec 18, 2009.

Thread Status:
Not open for further replies.
  1. King Grub

    King Grub Registered Member

    Joined:
    Sep 12, 2006
    Posts:
    814
    Article:

    http://blogs.pcmag.com/securitywatch/2009/12/av-testorg_releases_real-world.php
     
  2. vijayind

    vijayind Registered Member

    Joined:
    Aug 9, 2008
    Posts:
    1,413
    PC Tools is the real stunner here ... Guess the infusion of Threatfire in PCT IS 2010 has yielded rewards.
    Bitdefender is the biggest disappointer. Pushed to the bottom rug !!

    Rest of the result is on expected lines. A little disappointed by McAfee. I expected their Cloud-Detection to be better.

    This is probably the best test I have seen. Real world scenario with large sample set distributed over timeline. Kudos to AV-Test.org for this labour intensive test.:thumb:
     
  3. icr

    icr Registered Member

    Joined:
    Sep 6, 2008
    Posts:
    1,588
    Location:
    Mumbai
  4. lordraiden

    lordraiden Registered Member

    Joined:
    Jan 30, 2006
    Posts:
    3,067
    Norton and their free security products, PCToold are the winners
    Which was the AV engine of pctools?
     
  5. vijayind

    vijayind Registered Member

    Joined:
    Aug 9, 2008
    Posts:
    1,413
    PC Tools licenced VirusBuster engine and has modified it for their use and DB.
     
  6. InfinityAz

    InfinityAz Registered Member

    Joined:
    Jul 23, 2005
    Posts:
    828
    Location:
    Arizona
    I hope this is the beginning of a trend toward more real-life testing of security suites.
     
  7. Pleonasm

    Pleonasm Registered Member

    Joined:
    Apr 9, 2007
    Posts:
    1,201
    This test is very impressive: a 60-day evaluation of real-world performance.

    Two observations...
    • The alignment of the findings of this test to the similar one funded by Symantec and conducted by Dennis Technology Lab further weakens the argument that Symantec “influenced” the outcome of that test.
    • What is most surprising in this test, in my opinion, is the magnitude of the performance gap between the top performers (e.g., Norton, Kaspersky, PC Tools) as compared to the remainder (especially Avira, McAfee, CA, F-Secure, BitDefender and Trend Micro).
     
  8. vijayind

    vijayind Registered Member

    Joined:
    Aug 9, 2008
    Posts:
    1,413
    http://www.darkreading.com/security/antivirus/showArticle.jhtml?articleID=222002625

    Looks like the knifes are out !!!
     
  9. Pleonasm

    Pleonasm Registered Member

    Joined:
    Apr 9, 2007
    Posts:
    1,201
    Readers of this thread may be intersted in further published commentary on this test...

    However, NSS Labs has its own perspective to add...

    The comments by NSS Labs (even though they are a well-respected organization) may be dated, since it does not appear that they have tested the 2010 editions of the anti-malware products (see here).
     
  10. Fajo

    Fajo Registered Member

    Joined:
    Jun 13, 2008
    Posts:
    1,812
    It's good to see this kind of a test. It's long over due and nice to see the shift toward this the numbers are kind of what I expected. It shows who has been improving the product not just the detections I hope to see more of these in the future.
     
  11. Technic

    Technic Registered Member

    Joined:
    Aug 31, 2005
    Posts:
    428
    Something fishy there. Norton/Symantec/PcTools topping. :ninja:

    I trust AV-Comparatives.org more. :ouch:
     
  12. aigle

    aigle Registered Member

    Joined:
    Dec 14, 2005
    Posts:
    11,047
    Location:
    Saudi Arabia/ Pakistan
    I don,t know about the details but the very good result of PC tools may be due to the HIPS component of their fire wall.
     
  13. Fajo

    Fajo Registered Member

    Joined:
    Jun 13, 2008
    Posts:
    1,812
    People have such a hard time believing Norton being on top. I don't see why its so hard to believe they have more Market share, Manpower, Resources and Time in this business. Who knows maybe it's just biased is what blinds people. o_O
     
  14. Technic

    Technic Registered Member

    Joined:
    Aug 31, 2005
    Posts:
    428
    Norton/Symantec is ok. Just tell more about PC Tool's engine.

    I have tried them all...almost. :p
     
  15. Billy Blaze

    Billy Blaze Registered Member

    Joined:
    Mar 12, 2005
    Posts:
    79
    Location:
    Vorticon VI
    I think that is whats important to note, and something that I wonder if Rick Moy's comments took into account, or if in fact his comments are dated (as Pleonasm suggests) or taken out of context. However he is later quoted as saying "Focusing on just 10 zero-days does not reflect the current threat spectrum on the Internet."
    Which seems to suggest he is talking about the current AV-Test, but that he might be mistaken as I believe the total test set is 600.

    As quoted from the original link...
    "By contrast, tracking the 600 malware samples and 400 clean files took all of AV-Test's resources."

    "The current test simply evaluated whether the product detected each malware sample, without regard for which security component handled the detection."

    Considering all the resources used to conduct this test, I think it would have been some what interesting if a more detailed report was made available (perhaps separately) that included what security component of the AV Suite blocked the malware attack. Or at least a simple distinction between whether the antivirus component blocked the attack or other component of the suite.
     
  16. Fuzzfas

    Fuzzfas Registered Member

    Joined:
    Jun 24, 2007
    Posts:
    2,753
    I find the whole testing business highly entertaining. After having us bombarded for years with the "on-demand" tests, now they say to forget all about what we know , because now they do REAL WORLD tests. Which, by coincidence, have brought upside down the world of "antivirus detection" as we knew it. :D


    So basically, everyone that was saying that CA is crap, has now to think again, because it beats Bit Defender and F-Secure which were "top dogs" before. And Avira should hand the crown to... PC Tools! :argh:

    I love AV tests! They bring life to a forum! Within 5 months, everyone will have forgotten of the "old kings" and will be accepting the new natural order of things.

    - Before: "I have the best of the best: Avira Free! AVG? Does it even detect anything?"
    - After 5 months: "Dude, don't get Avira! It's decent at best, but that's all. Get PC Tools or AVG Free, the best freebies"! :)

    Until some other "more real life" test method comes out which will shatter our realities once again. :D

    I 'd say pick an AV you like, do some of your own tests and be happy. AV tests are a relative strength indicator for me and nothing more.
     
    Last edited: Dec 18, 2009
  17. Fajo

    Fajo Registered Member

    Joined:
    Jun 13, 2008
    Posts:
    1,812
    The main thing I do like about this test. It tests the product as a whole not just one portion of it. Makes it so you can see what your product really does protect against.
     
  18. Page42

    Page42 Registered Member

    Joined:
    Jun 18, 2007
    Posts:
    5,828
    Location:
    Last Breath Farm
    I agree... I like that it tests all aspects of a product.

    What I find unusual is that with 10 fresh threats introduced each day for 60 days, it sounds as though the systems are not cleaned after each day, so an infected system (or a completely healthy one, depending) is essentially attacked repeatedly. If a threat goes undetected, it seems like the compromised machine would be crippled to some extent, and even more prone to further infection. I don't know if clinically that is correct, but it sure does emulate real world conditions.
     
  19. dawgg

    dawgg Registered Member

    Joined:
    Jun 18, 2006
    Posts:
    817
    Quick graph making it easier to see what the AV-Test statistics mentioned in the initial post is showing:
    block.PNG detection.PNG


    What is the difference between the "detection" and "blocking"? Shouldn't detection = block?
     
  20. Page42

    Page42 Registered Member

    Joined:
    Jun 18, 2007
    Posts:
    5,828
    Location:
    Last Breath Farm
    Free versions of Prevx detects, but does not block (most threats). Paid version of Prevx protects, or blocks.
     
  21. Pleonasm

    Pleonasm Registered Member

    Joined:
    Apr 9, 2007
    Posts:
    1,201
    That seems as likely an explanation as any....

    Bias is defined as “a particular tendency or inclination, esp. one that prevents unprejudiced consideration of a question” (source: dictionary.com).

    Yes, looking at the protection provided by the anti-malware product in total under real-world conditions logically seems to be the most accurate way to assess performance.

    In my opinion, this test by AV-Test is now the "gold standard" against which all other anti-malware comparatives must be judged.

    An interesting question....

    Nonetheless, if you look at the paper by AV-Test entitled Testing of “Dynamic Detection” (available here), the ideal test scenario is described as: “Record the actions of the security software, and compare this record to actions of the clean base system.” The phrase “clean base system” in this paper seems to suggest that AV-Test would have restored a backup image to the PC following the introduction of each tested malware threat.

    The same paper also states "The clean state of the operating system has to be known" as a prerequiste for testing.
     
  22. Fajo

    Fajo Registered Member

    Joined:
    Jun 13, 2008
    Posts:
    1,812
    This is for Suites only. Last I heard Comodo was staying out of tests until they felt there AV was on par.


    Edit.
    The person I was quoting has ether deleted his post or been removed by a mod. So if it seems out of place that is why.
     
  23. Ibrad

    Ibrad Registered Member

    Joined:
    Dec 8, 2009
    Posts:
    1,949
    Interesting test, interesting results :D

    I find it surprising that PC Tools did well. Very glad Panda did well :D
     
  24. Page42

    Page42 Registered Member

    Joined:
    Jun 18, 2007
    Posts:
    5,828
    Location:
    Last Breath Farm
    But that doesn't mean that this test was done that way, does it? All this current test says is... "Each day for 60 days, researchers released 10 fresh threats on the test systems and analyzed each product's ability to detect the threat and to fully block its installation." To me, that implies that the systems were left intact. I'd like to know for sure.
     
  25. Pleonasm

    Pleonasm Registered Member

    Joined:
    Apr 9, 2007
    Posts:
    1,201
    Note that the reported “detection rates” also include “warning messages.” If a user is warned that a download is potentially malicious and still executes the file, the fact that the anti-malware product doesn’t block the threat (while undesirable) should not detract, in my opinion, from the assessment of the quality of the product. No product can protect against all “foolish decisions” by users, after all.
     
Loading...
Thread Status:
Not open for further replies.