The danger of AV testing sites

Discussion in 'other anti-virus software' started by Bodhitree, Dec 20, 2012.

Thread Status:
Not open for further replies.
  1. Bodhitree

    Bodhitree Registered Member

    Joined:
    Dec 5, 2012
    Posts:
    567
    Anyone else see a bit of danger with these testing sites?

    What I mean is, some folks I know base their entire decision year after year on the test results of these sites. Ignoring the fact that they are synthetic tests, and ignoring the fact that these tests do not take into account the most important aspects of a product. I think this is kind of reckless but these people don't know any better. For example I had to spend hours explaining to someone why a product with top signature scores on AVC won't protect as well as a product with an exceptional HIPS and Website Filtering. He just couldn't wrap his brain around it, and ultimately used the synthetic test to motivate his purchase decisions, which I feel is a mistake.

    For example AVC hasn't tested BG2013 yet, which is a remarkably improved product over 2012. But AVC will run a blind signature test, ignoring the fact that BG uses Commtouch resources for some seriously powerful HTTP malware blocking. This won't reflect properly on these tests. Comodo has a great HIPS, but always seems to score lower on synthetic signature testing, but everyone knows it's got a fantastic HIPS. Webroot is another story, Webroot is strong in my experience (and testing), but only because it is strong in real world protection. But if you take Webroot and toss 2000 malware files at it and expect it to do well then it will not. It cannot analyze 2000 files that fast on the cloud, but if you try to execute any of those - Webroot will kick in.. These synthetic tests ignore what is important, and I feel at times - mislead people.

    Am I wrong about this? Or are these tests in some cases, doing more harm than good? I tend to ignore them, because other aspects are more important to me. But I know a lot of folks out there use them like some kind of product bible.
     
  2. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    7,927
    Location:
    The land of no identity :D
    Re: The danger of sites like AV Comparatives.

    AV-C, for example, performs multiple different types of tests and even explicitly states that these tests are not the only criteria one should use to make a decision on purchashing an anti-malware product.

    I think the scenarios across all decent tests today (AV-Test, AV-comparatives, even MRG) are sufficient enough as real world scenarios. Signature based detection is still important to an extent (cloud or not).

    People base their decisions off mainly the on-demand scores because; let's face it....an average guy is going to think the product that reported "Trojan/Phenomic" is obviously better than the product that detected "possible malware". On top of that, alert based detections give a lot of room to make errors on the part of the user.

    Now, it's up to you where your priorities lie - personally I take a mix of everything. Things like customer support and convenience to the licensee matters to me. Thus, most products I use are average-to-good in most tests; but I'm not really worried about that - as long as they still pass, they are good enough for real-world usage - but the customer service makes a hell of a difference when you do encounter trouble :D
     
  3. jack76

    jack76 Registered Member

    Joined:
    Jul 3, 2009
    Posts:
    72
    Location:
    Helvetin Kylm
    Re: The danger of sites like AV Comparatives.

    Bullguard 2013 is on AV-C real world protection test
     
  4. Bodhitree

    Bodhitree Registered Member

    Joined:
    Dec 5, 2012
    Posts:
    567
    Re: The danger of sites like AV Comparatives.

    I know, and it scored Advanced+++ . I need to read about this test, was it 'really' real world, as in surfing the web and hitting Malware sites? I would guess it was simply a test of actually firing off malware and seeing how a product reactions. That's a good test, but still ignores extremely important HTTP protection.
     
  5. jack76

    jack76 Registered Member

    Joined:
    Jul 3, 2009
    Posts:
    72
    Location:
    Helvetin Kylm
    Re: The danger of sites like AV Comparatives.

    AFAIK it is surfing and hitting malware sites...just look at Trendmicro`s results
     
  6. safeguy

    safeguy Registered Member

    Joined:
    Jun 14, 2010
    Posts:
    1,709
    Re: The danger of sites like AV Comparatives.

    I get your point but I don't quite agree on calling it a 'danger'. Reports are just that - statistics displayed for those interested. Forget AV testing, this applies to any testing results for anything else.

    If a user is willing to base a decision solely on a particular test result while ignoring the rest, it's ultimately his/her call. One may want to read a report and take things at face value while others may analyze things deeper. There's no right or wrong here.
     
  7. Baserk

    Baserk Registered Member

    Joined:
    Apr 14, 2008
    Posts:
    1,317
    Location:
    AmstelodamUM
    Re: The danger of sites like AV Comparatives.

    Perhaps the easiest way to explain the difference between the real-world and on-demand test scores, is to still use a car test analogy.
    I often explain that an on-demand test, only tests 'the car engine', a real-world test also measures suspension, stiffness, gearbox, weight etc.
    Such analogies are flawed, I know, but ime most folks immediately understand the 'full car' test vs 'engine-only' test analogy.

    It's usually the security software companies who will use on-demand test results and portray them as real-world results or simply use blanket statements as in 'highest detection, winner of gold medal blabla'.
    AV-Comparatives are not to blame for misinterpreting or creative use of test results. Only a minority will read the full test report offering detailed info for every test.
     
    Last edited: Dec 20, 2012
  8. Macstorm

    Macstorm Registered Member

    Joined:
    Mar 7, 2005
    Posts:
    2,531
    Location:
    Sneffels volcano
    Re: The danger of sites like AV Comparatives.

    I think I'll always like their old school approach, pure file-detection tests, don't care much on some negligible false positives rate:
     
  9. larryb52

    larryb52 Registered Member

    Joined:
    Feb 16, 2006
    Posts:
    1,126
    Re: The danger of sites like AV Comparatives.

    I read them with a grain of salt , I use what I feel will keep me safe for what I do. The way I see it is there no test that will do what you do on a computer day in day out. Even the best of programs miss something if you keep your browser in areas of the internet that are prone to viruses, Or if you click on a bad link as you just ok'd you computer to go there. So these tests are just that tests not meant to be nothing but guides...
     
  10. The Hammer

    The Hammer Registered Member

    Joined:
    May 12, 2005
    Posts:
    5,619
    Location:
    Toronto Canada
    Re: The danger of sites like AV Comparatives.

    I'm glad someone else around here appreciates HTTP protection mechanisms.
     
  11. JerryM

    JerryM Registered Member

    Joined:
    Aug 31, 2003
    Posts:
    4,221
    Re: The danger of sites like AV Comparatives.

    While I do not consider results of AV-C tests the only criterion, I place a high value on those results.
    Don't see any danger or reason to change.
    Jerry
     
  12. Brandonn2010

    Brandonn2010 Registered Member

    Joined:
    Jan 10, 2011
    Posts:
    1,849
    Re: The danger of sites like AV Comparatives.

    I don't think AV-Comparatives would be as popular as it is if it wasn't reputable and trustworthy. Between it and AV-Test, I think they provide an accurate picture on how effective AVs are.
     
  13. Macstorm

    Macstorm Registered Member

    Joined:
    Mar 7, 2005
    Posts:
    2,531
    Location:
    Sneffels volcano
    Re: The danger of sites like AV Comparatives.

    Don't worry people, the AV of your liking will be tested at av-c again someday... hopefully :D
     
  14. ronjor

    ronjor Global Moderator

    Joined:
    Jul 21, 2003
    Posts:
    57,752
    Location:
    Texas
    Re: The danger of sites like AV Comparatives.

    Some off topic posts removed.
     
  15. Notok

    Notok Registered Member

    Joined:
    May 28, 2004
    Posts:
    2,969
    Location:
    Portland, OR (USA)
    Re: The danger of sites like AV Comparatives.

    I don't think that the tests are a danger in any way, but they certainly don't show the bigger picture. You can also never know just how it will translate to protection for you.

    Every security solution is made with a particular strategy in mind, and the current tests don't provide enough info to get much insight as they used to when they would give a detailed breakdown of the types of malware samples being tested.

    It would be particularly useful with the new trend of cloud AVs that are compatible with other solutions. Maybe we know that two of them catch 80% each, but there's no sense of whether they're detecting the same things or if they truly compliment each other, and so adding more than one layer seems like a guessing game. It would be nice to know where the strengths and weaknesses of each lie.

    To me it just seems like we don't have enough info to get a grasp of the bigger picture. That's not dangerous, per se, but it makes it hard to devise a strategy beyond choosing a brand.

    And as an aside about "classic" HIPS; I remember when Prevx was a pure behavior blocker, without intelligence, and they found that up to 80% of users were allowing a popular worm at the time. That was early on when the majority of users would be enthusiasts, and so you can't automatically assume that it wouldn't happen to you; when a product puts the security in your hands, you can't assume a 100% detection rate any more than you can with any other solution (especially considering that many of these are meant to fool people that deal with this stuff frequently as part of their job).

    No matter how diligent you are, I would say that if some malware is running, and completely missed by scanners, then you have to assume that your system is compromised no matter how many times you clicked Block. After all, if I were running a keylogging blocker and found an active banking trojan running, I wouldn't assume that the blocker worked and all is fine -- I'd take the same actions, either way, of changing passwords and calling the bank.

    So I'd rather have something that reliably blocks 80% of threats, than something that is a complete unknown but depends largely on my own state of mind.

    I know that's a bit of a tangent, but the problem I see is in the areas of ambiguity. I agree that the tests aren't enough, and they state as much themselves, but OTOH they're about all we have to go on. I just hope that we can get some more detailed information about these tests in the future so that we can fine tune our own strategies a little better.
     
    Last edited: Dec 21, 2012
  16. century

    century Registered Member

    Joined:
    Oct 13, 2007
    Posts:
    92
    Re: The danger of sites like AV Comparatives.

    Ordinary people, the novices, will read the test reports, compare the results & make a decision based on these results. What else they can do?
    Most of the members of this forum also seem to be influenced by the test results. And I find nothing wrong in that. The tests act as guides only. But how many people who use computers can have any idea about "real world", "zero day" and stuff like that? They must go by the tests. Common sense of these novices, myself included, dictates that one should put his money in a software that performs consistently well in the tests & suppliment it with prayers. Because no security software can guarantee hundred percent protection.
     
  17. er34

    er34 Guest

    Re: The danger of sites like AV Comparatives.

    I agree with you. I know many people who ask me "is it true that AV1 is better than AV2, because I read it on <testing org site>". I tell them "No it isn't because..." and they answer "But how? It is written on <testing org site>, then it must be 100% true" and so on...
     
  18. BoerenkoolMetWorst

    BoerenkoolMetWorst Registered Member

    Joined:
    Dec 22, 2009
    Posts:
    3,766
    Location:
    Outer space
    Re: The danger of sites like AV Comparatives.

    Every piece of information can be dangerous when the one obtaining the information misinterprets it and then acts on his/her view of it.

    Then perhaps should read the test report as the samples are downloaded from real URLs so a product with HTTP scanning/URL blocker has every chance to use it.
     
  19. TairikuOkami

    TairikuOkami Registered Member

    Joined:
    Oct 10, 2005
    Posts:
    2,509
    Location:
    Slovakia
    Re: The danger of sites like AV Comparatives.

    Not really, since they are mostly meant for IT enthusiasts and for managers of big companies, who love to buy products with the highest results.
    99,99% people never heard of it and more knowledgeable people from Wilders take it with certain reservations, when choosing products.
    Average people choose products randomly, at best they ask in a forum, where they get 100 opinions from 100 other average users.
     
  20. PJC

    PJC Very Frequent Poster

    Joined:
    Feb 17, 2010
    Posts:
    2,959
    Location:
    Internet
    Re: The danger of sites like AV Comparatives.

    If some fanboys are Not happy with the results of their "favorite" AV
    in the Tests of AV-Comparatives and AV-Test,
    this does Not mean that AV-Comparatives and AV-Test
    are Not Reliable AV Testing Organizations.
     
  21. Fuzzfas

    Fuzzfas Registered Member

    Joined:
    Jun 24, 2007
    Posts:
    2,753
    Re: The danger of sites like AV Comparatives.

    I really like the "performance test" in AV-Comparatives. It's the most "certain" test out there and agrees with empirical use.
     
  22. wtsinnc

    wtsinnc Registered Member

    Joined:
    Oct 3, 2008
    Posts:
    943
    Re: The danger of sites like AV Comparatives.

     
  23. silverfox99

    silverfox99 Registered Member

    Joined:
    Jul 14, 2006
    Posts:
    204
    Re: The danger of sites like AV Comparatives.

    Not sure i agree with 'danger' as such. Each test is informative on a particular aspect of protection or disinfection, system load etc and the best way is to look at trends and results over a longer period eg 6 months+ or the whole year. We're probaly agreed on that.

    Look at Trend Micro amazing results recently on AV-C & AV-T but that's only in the last few months. It's not in my 'top tier' yet of AVs that are consistent over longer periods in a variety of test organisations tests (AV-C, AV-Test, Matousec, MRG, Virusbtn etc). For the last 12 months for example BD, Kaspersky, F-Secure, BG, G-data have all performed consistently well across a variety of tests by a number of test organisations.

    That said - you can get 6 months free Trend IS just now at Facebook AV Marketplace. Tempting to give it a whirl.
     
  24. er34

    er34 Guest

    Re: The danger of sites like AV Comparatives.

    Nobody here has ever said that these organizations are unreliable. It is just their methodologies that must be seriously improved to reflect absolute real-world situations. They test all products under same condition - in reality this can't happen and is wrong. And the final mark certain product gets must be not only detection and usability, but many... many other factors
     
  25. kareldjag

    kareldjag Registered Member

    Joined:
    Nov 13, 2004
    Posts:
    622
    Location:
    PARIS AND ITS SUBURBS
    Re: The danger of sites like AV Comparatives.

    the word danger is inappropriate, and i would replace it by "bad consumerist impact".

    -most popular av organizations tests do not help the average user to choose the most appropriate product, in relation to his experience, location and language, machine performance, privacy and support perspectives...
    The rule is here done by computers mags.

    -many av tests organization, especially VB100 archieve 100%.
    But primo it has been demonstrated in a mathematical manner that blacklist detection could not reached this result, and Segundo, the history of Insecurity has shown that antivirus software have failed.
    This vicious marketing impact gives a false sense of absolute security in user mind.
    Yes of course, buy a Slendertone product and you ll get a Mr Olympia body, and buy an VB100 certified av and your computer will be immune to malwares.

    More over, some tests organization are ethically corrupted by default due to interest conflicts: some of them are sponsored by some av campanies (mostly Symantec), and some other require money participation from av editors (AV-Comparatives).

    Preaching in the desert since years on this board, but objective statements must be said.

    Rgds
     
Loading...
Thread Status:
Not open for further replies.