The danger of AV testing sites

Discussion in 'other anti-virus software' started by Bodhitree, Dec 20, 2012.

Thread Status:
Not open for further replies.
  1. Mman79

    Mman79 Registered Member

    Joined:
    Sep 19, 2012
    Posts:
    2,016
    Location:
    North America
    Re: The danger of sites like AV Comparatives.

    Completely off topic, but I've found through *ahem* "scientific" research that the majority of the more well known porn sites are surprisingly clean (with ad-blocking in place), and so are many P2P websites. A few years ago, they would be the first place you'd get "pwned". Now? I see more risk on personal blogs, search engines and popular media streaming websites, and not just the illegal type.

    You know what you're doing, so these testing procedures are fine for you, but handing out that advice to just anyone is asking for trouble.
     
  2. gkweb

    gkweb Expert Firewall Tester

    Joined:
    Aug 29, 2003
    Posts:
    1,932
    Location:
    FRANCE, Rouen (76)
    Re: The danger of sites like AV Comparatives.

    Hello,

    I think that sites such as av-comparatives and av-test are doing an invaluable work. They are a giving an accurate picture of AV effectiveness, even if there is still new tests needed to be done (e.g on 64 bits Windows).

    For instance last time I wondered of AV effectiveness on mobile/tablet, they did already test them with published results.

    It doesn't prevent us to keep an open mind and a skeptical/objective thinking, of course.

    Just my opinion.

    Regards,
    Guillaume.
     
  3. Noob

    Noob Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    6,491
    Re: The danger of sites like AV Comparatives.

    The car analogy fits this thread well.
    When you purchase a car . . . you look into specs, engine, quality and especially security.
    How do you know a car is safe?
    By looking at crash tests . . . even though you are not expecting to crash you always go for a car that has a high score just in case because while you may be a good driver others may not be so good at it and mistakes happen. :D

    The same thing applies to AV tests.
     
  4. The Red Moon

    The Red Moon Registered Member

    Joined:
    May 17, 2012
    Posts:
    4,101
    Re: The danger of sites like AV Comparatives.

    That is a rather bizarre analogy.o_O
    If a computers security gets breached then its not the end of the world but unfortunately if a car is unsafe then it could cost a life so is there no comparison.o_O
     
  5. Mman79

    Mman79 Registered Member

    Joined:
    Sep 19, 2012
    Posts:
    2,016
    Location:
    North America
    That's because you read too much into it. Granted there are better analogies, but the point is that (though I don't necessarily agree it's best for normal users) you look at everything an AV provides (the "anti-lock brakes", "power steering" of the digital world), you look at its "crash performance" (did the thing get buried by particular types of malware or did too much bypass it?), quality (does it use the latest technology, does it run well?).
     
  6. PJC

    PJC Very Frequent Poster

    Joined:
    Feb 17, 2010
    Posts:
    2,959
    Location:
    Internet
    The Number of Different Malware
    an Average user can deal with
    each month is what? 20? 50? 100? at the most.

    This Number is, by FAR, Less than
    the thousands of different Malware
    AV-C and AV-Test deal with.

    This is much more Realistic than
    the 'Real-World' "Testing"
    of some individual "Testers" who Lack both
    AV-Testing Expertise and Infrastructure.
     
  7. Notok

    Notok Registered Member

    Joined:
    May 28, 2004
    Posts:
    2,969
    Location:
    Portland, OR (USA)
    Indeed; well said :) (even if the U and B tags do make my head hurt :D )

    The professional tests are done by knowledgeable experts that know what the threat landscape looks like, how the threats work, and how to test in the most meaningful manner.

    A relatively small test with a small sample size and no particular methodology is pretty much meaningless, because there's too much room for coincidence and other confounding factors. Just because a product works well against a couple dozen sites doesn't mean that it will do well against the next couple of dozen, especially if they're run by different factions of criminals that use different malware. An AV's detection may also wax or wane at the next crop of threats, especially when the attackers are replaced by another group, develop a new strategy of their own, and/or deploy the next new threat.

    Security depends on coming up with a good security strategy. While AV tests don't always reflect real-world performance, they are one thing among many to consider when developing a strategy -- particularly detection rates over time. If you're asked to help someone else and you just install a product with your favored brand name after using it (even heavily) for a week or month, then you're not doing anything for them they couldn't do themselves when the next door neighbor kid tells them that Brand X kept him safe and Brand Y sucks (even worse when the kid's opinion is partially developed from unnamed people on internet forums about computing, that each inevitably derive their opinions from someone that works with computers and has "real world experience" in the matter).

    It's easy to abstract these tests and trivialize them with terms like "synthetic" and "honeypot threats," but unless you have intimate knowledge of the threat landscape and the testing methodology used (e.g., not all honeypots are the same, and they may be doing the same thing you're talking about but with many more websites), it would be foolish to dismiss them and even more foolish to do so in a way that implies that you somehow know better than the people that devote their waking lives to the subject matter. (But hey, at least "synthetic" sounds sciency!)

    There's nothing wrong with saying "I don't know" (unless you're in a job interview or taking an exam), but it's another matter to (implicitly or explicitly) declare your own knowledge and testing to be superior.
     
    Last edited: Jan 1, 2013
  8. People are having a CRY just because their favorite child scores poorly on a test. I know if my kid scored poorly on a test I'd be upset so it's only natural to come up with excuses.

    Like the test is not done correctly, run by amateurs etc But in fact it's just our emotional investment in the product at play, no one wants to buy a lemon. Worse yet is someone telling you bought a lemon.
     
  9. The Red Moon

    The Red Moon Registered Member

    Joined:
    May 17, 2012
    Posts:
    4,101
    Yes but sometimes you have to admit that your child may not be as bright as others and thus will always fair badly at tests.:argh:
     
  10. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    8,251
    Location:
    The land of no identity :D
    Know that a great many nobel laureates really were no good at school/university. Calling them "not as bright" is an injustice. :)

    I'm sorry, but I just had to say this.

    A further analogy: I may put huge efforts to raise my test score in physics, but that doesn't mean I am a good physicist - it just means I know how to crack the test ;)

    Thing is, it's subjective - if you think something works well for the kind of scenario you are using it in, then by all means you cannot call it inferior. The tests are there for reference, they are never an absolute measure :)
     
  11. Noob

    Noob Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    6,491
    Re: The danger of sites like AV Comparatives.

    Oh My Gad, that was so funny, maybe i find it funny because . . . well, i don't need to say much. :ninja:
    Hahahahaha
     
  12. PJC

    PJC Very Frequent Poster

    Joined:
    Feb 17, 2010
    Posts:
    2,959
    Location:
    Internet
    It happens all the time...
    Poor Test Results? Let's blame it on the Tester...
    Certainly, a Convenient Excuse...
    That's the Bitter Truth, and it takes a lot of Courage to admit it.
     
  13. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,592
    Location:
    U.S.A.
    I guess you don't use WOT when surfing porn sites? Very few of these sites are fully trustworthy.
     
  14. xxJackxx

    xxJackxx Registered Member

    Joined:
    Oct 23, 2008
    Posts:
    8,623
    Location:
    USA
    And how trustworthy is WOT in these situations? How many conservative types mark these sites bad just to keep others out? If I were to do any surfing that I expected to be risky, I would start up Sandboxie.
     
  15. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    8,251
    Location:
    The land of no identity :D
    I am saying it again: This is entirely subjective. Many, many people are out there who fared much better in the real world than in tests. It's similar with AV products; some really do seem to be better in the real world than in tests.

    The real bitter truth is what I stated: tests are a reference point and simply cannot account for every single paradigm of working. As a result, they are a reliable reference but not an indicator of suitability for one's personal tastes :)
     
  16. Bodhitree

    Bodhitree Registered Member

    Joined:
    Dec 5, 2012
    Posts:
    567
    Re: The danger of sites like AV Comparatives.

    Fixed that for you.

    Also, didn't AVC refuse certification, even though Comodo offered to pay the $50,000 fee to get them inspected, and certified? AVC also said they were in the process of certification, allegedly - and that was over a year ago.. That's an interesting, and ongoing subject.

    http://www.melih.com/2011/12/07/com...omparatives-org-for-test-auditing-validation/

    People have to be hold accountable John. Where would this world be if there was no accountability? We all have to work on keeping everyone honest. These av comparatives guys claimed they were getting ISO accreditation to our user base a year ago. Did they? Our users must be given the truth and we must not let people provide misinformation. We always protect our users! -Melih
     
  17. m00nbl00d

    m00nbl00d Registered Member

    Joined:
    Jan 4, 2009
    Posts:
    6,623
    Re: The danger of sites like AV Comparatives.


    I'm a bit lost. Are you Melih, or are you quoting something from that URL? I'm asking because I'm not seeing that text there... :doubt:

    But, anyway... this kind of tests has some odd effect on a lot of people, and that's the fact that it will make these people switch their antimalware solution every time a new test comes out. Even if these tests would reflect the real world, the same wouldn't necessarily mean they would protect the users now as good as they protected in those tests before. Crazy world... o_O
     
  18. Fuzzfas

    Fuzzfas Registered Member

    Joined:
    Jun 24, 2007
    Posts:
    2,753
    Re: The danger of sites like AV Comparatives.


    Maybe he is quoting some post from this:

    http://forums.comodo.com/melihs-cor...-pay-50000-to-avcomparativesorg-t79151.0.html

    I don't know, i am still on page 1, but just guessing, since it's the "talkback" from that blog post.

    Although it would be really cool if he was Melih. :D


    My take on AV tests is: If you really believe that the AV is effective 99,99%, you are free to, but this would also mean that virtually no infected PCs should exist. For me AV tests are good to show relative(not absolute) strength and the more testers the merrier. Meaning, if in say 4 different tests, AV1 ranks above AV2, then chances are that it's a safe conclusion that it is better.
    Certainty? No. Because none of the test i have seen is statistically structured to reflect "real world".

    But then you have for example one lab (for example, sponsored by an AV company) that oddly enough ranks another product 1st. Well, still, if the other top dogs are the same as in the other tests, again, there is a good chance they are really top dogs.
     
    Last edited: Jan 5, 2013
  19. er34

    er34 Guest

    :) :thumb: :) Great, well said , Firecat!
     
  20. Fuzzfas

    Fuzzfas Registered Member

    Joined:
    Jun 24, 2007
    Posts:
    2,753
    To Melih's defence (did i just say that?), if you take Comodo as a "suite", it's hard to see how it would fare worse than probably any other suite. Meaning, if you have D+ enabled with all bells and whistles, even if the AV doesn't detect everything, it's almost impossible to infect yourself. But, what test follows a methodology which would accept D+ blocking/sandboxing as "detection"?
     
  21. Baserk

    Baserk Registered Member

    Joined:
    Apr 14, 2008
    Posts:
    1,321
    Location:
    AmstelodamUM
    Re: The danger of sites like AV Comparatives.

    As per Melih's statement;
    'None of the AV-Comparatives.org tests are validated by a credible, independent 3rd party.
    It is in the end user interest to have the tests validated. There are many respectable independent and qualified organisations who can audit and validate AV-Comparatives testing methodologies and financial transactions.
    '

    You really expect an organization like AV-C to have it's financial transactions audited by a third party, just because someone is willing to pay for it?
    The offer implies there is actually a need to have not only AV-C tests audited but its finances also, top to bottom, inside and outside before accepting any of it's test results.
    A somewhat bizarre (and thus not entirely surprising) offer imo.
    Having a desire to have as much details about an organization's testing methodologies is one thing, suggesting there's a need for all it's financial details, implies malfeasance.
    I'm not surprised AV-C refuses to shake someones hand while being kicked against the shins at the same time.
     
  22. Bodhitree

    Bodhitree Registered Member

    Joined:
    Dec 5, 2012
    Posts:
    567
    Re: The danger of sites like AV Comparatives.

    Actually, yes I do. The reason for audits of financials for certification is simple - to prevent abuse, and to ensure specific payments or funding doesn't impact final results. This is crucial for certification, don't you want to know if a particular AV is giving them money? Transparency is the key, and certification helps ensure transparency.
     
  23. Noob

    Noob Registered Member

    Joined:
    Nov 6, 2009
    Posts:
    6,491
    Wow Melih really knows how to stir things up. :D :argh:
     
  24. Mman79

    Mman79 Registered Member

    Joined:
    Sep 19, 2012
    Posts:
    2,016
    Location:
    North America
    Re: The danger of sites like AV Comparatives.

    I agree with this. How many times has the same back and forth gone on about MS "sponsored" tests/reports? No, I'm not picking on MS, but that's a good example of the point Bodhitree is making.

    @Noob: The guy always has, lol.
     
  25. Fuzzfas

    Fuzzfas Registered Member

    Joined:
    Jun 24, 2007
    Posts:
    2,753
    Ah, you should have seen the times when Melih was member of the forum (i think he was at some point banned). The forum was so lively! It was like visiting Las Vegas compared to a cemetary. Good times... :D

    P.S.: We miss you, Melih! You could always come back registered as Helim or Mileh, i swear i won't tell!
     
Thread Status:
Not open for further replies.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.