New virus.gr comparative test of AVs

Discussion in 'other anti-virus software' started by tiagozt, Jun 25, 2008.

Thread Status:
Not open for further replies.
  1. jrmhng

    jrmhng Registered Member

    Joined:
    Nov 4, 2007
    Posts:
    1,268
    Location:
    Australia
    Professional virus testers like IBK and Marx have enough issues already dealing with some of the diffculties involved in malware testing. This tester is a virus collector. How do we know that his testbed is representative? How do we know that there isn't irrelevant zoo virui? How do we know that he has effectively cleaned out the rubbish?

    Well not too hard when you pile on a few AVs :p
     
  2. Macstorm

    Macstorm Registered Member

    Joined:
    Mar 7, 2005
    Posts:
    2,631
    Location:
    Sneffels volcano
    well it's not a secret, don't ya? ;)
     
  3. yxclark

    yxclark Registered Member

    Joined:
    Jun 15, 2008
    Posts:
    20
    IBK and Marx are "professional" only by the virtue of their reputation. After all, nobody here has access to their sample sets, so the best we do is get by via hearsay, while the fact is that the very same questions you ask about Virus.gr are equally valid for AV-Comparatives and AV-Test.
     
  4. Diver

    Diver Registered Member

    Joined:
    Feb 6, 2005
    Posts:
    1,444
    Location:
    Deep Underwater
    This guy has never been considered to be one of the respected AV testers. Just search around here for old threads on it.
     
  5. yxclark

    yxclark Registered Member

    Joined:
    Jun 15, 2008
    Posts:
    20
    That's my point exactly: hearsay, more hearsay, and yet more hearsay. Has anyone actually seen his sample set and confirmed them to be not statistically representative? I don't think so.

    Therein lies the problem with these antivirus tests. Not only do people rely on other people to lead them blindfolded when it comes to testing the performance of antivirus products, they also do the same when it comes to the reliability of those tests. It's a bunch of blind people groping about in the dark yet thinking they're competent enough to lead other people.
     
  6. bellgamin

    bellgamin Very Frequent Poster

    Joined:
    Aug 1, 2002
    Posts:
    6,825
    Location:
    Hawaii
    No doubt you are an ardent fan of Abacre AV.:cautious:
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    As to "professional" versus "amateur" anti-malware testers --

    By definition, a "professional is in it for the money. That does NOT mean that a pro necessarily does a better job than an amateur -- it just means that the pro insists on being paid for what he does.

    An "amateur" is from a Latin word for "love" (amo, amas, amat) -- the amateur does what he does for the love of doing it.

    To wit -- Tiger Woods was a superb golfer as an amateur. Now he is a pro. Being a pro doesn't make him a better golfer, just a richer one.
     
    Last edited: Jun 25, 2008
  7. pandlouk

    pandlouk Registered Member

    Joined:
    Jul 15, 2007
    Posts:
    2,964
    What do you mean professional virus testers? What is it a new terminology?

    As yxclark stated actually none has seen their testbeds either (unless I am mistaken). So how can you be sure that they have cleaned out the rubbish?

    I already have, and although some of his earlier tests were more flawed than those of AV-Comparatives and AV-Test, he learned from his mistakes and made better his methothology.

    ps. I have more respect for the tests made by a hobist. At least he does not get funds (yet :p ) from antivirus companies. Most of those well respected tests evolved in a great publicity system. :cautious:
     
  8. L815

    L815 Guest

    There is no one better volunteer to test viri accurately, but thyself.
    :isay:
     
  9. jrmhng

    jrmhng Registered Member

    Joined:
    Nov 4, 2007
    Posts:
    1,268
    Location:
    Australia
    You trust Checkpoint firewalls because of their reputation but you don't have access to their source code. You trust PGP because of their reputation but you don't have access to their source code. You trust PwC as auditors because of their reputation, but you don't have access to their audit methodology. You trust your doctor because of his/her reputation but you dont know what it is that they are doing. The list goes on. They best we do with all these people/firms is by hearsay.

    The questions I ask are equally valid for every virus tester however at least you know that for avcomparatives and avtest, there is a debate, their methodology is being challenged, there is due process. You cant say say that about virus.gr.

    That is in the context an amateur/professional dichotomy. My use of the word profession suggests that it is an occupation that requires training and or qualifications. Malware testing is difficult and so someone needs to demonstrate that they have a minimum level of skill and understanding of malware before I trust their tests.

    For example

    This seems to make little sense and only suggests that his sample, due to it's size, is representative of the population of viri. While this makes sense in a statistical sense (i.e. this sample is large enough to be representative of the population) it shows little consideration of why sampling needs to be non random. The questions on test bed selection have not been answered.
     
    Last edited: Jun 25, 2008
  10. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    8,183
    Location:
    The land of no identity :D
    Its a mistake.....he probably just copied the notes from his previous tests when Avira and AntiVir used to be separate products.
     
  11. Diver

    Diver Registered Member

    Joined:
    Feb 6, 2005
    Posts:
    1,444
    Location:
    Deep Underwater

    Are you saying this guy is perfect until proven otherwise? Its the expert that has the burden of proving his credentials before he may offer an opinion, since you appear to be stuck on legal analogies. The issues regarding his sample set have been analyzed more than once. Of course no one around here can go in and audit it, so judgments must be made from the available information. And finally, your view that references to any relevant prior discussion is hearsay is just plain inappropriate.
     
  12. yxclark

    yxclark Registered Member

    Joined:
    Jun 15, 2008
    Posts:
    20
    No, I'm just pointing out the interesting inconsistencies that a gullible community exhibits. AV-Comparatives and AV-Test are "professional" and Virus.gr is "suspect", when the fact is that we know equally much, aka nothing at all, about how they conduct their tests.

    Or maybe perhaps you're privy to some exclusive insider information?
     
  13. jrmhng

    jrmhng Registered Member

    Joined:
    Nov 4, 2007
    Posts:
    1,268
    Location:
    Australia
    I quote myself

    We do know more about avcomparatives and avtest than we know about virus.gr. Yes you have a point in that we do rely on other signals such as reputation, branding etc. However that is true in every situation where there is information asymmetry. Again, think about why you trust your doctor when you are sick, why you trust the auditor as you read a financial statement. You know very little about what they are doing (unless you are in fact a doctor or auditor) but you trust them because of their reputation and branding.

    However, in this case, not only are we relying on these signals, we also see the due process going on with avcomparatives and avtest that we just cannot observe with virus.gr. In fact there are many signs that the test was flawed. Consider

    So programs were left out because he couldnt get them to work? Isnt that an issue?
    There is obviously a difference between the two AVG products as one has a spyware engine. He considers them to be the same.

    Also consider

    So the sampling was based on an AV detecting the virus? Doesn't this cause significant bias against the programs not mentioned?
     
    Last edited: Jun 26, 2008
  14. yxclark

    yxclark Registered Member

    Joined:
    Jun 15, 2008
    Posts:
    20
    Well, if you can't get a program to work, then obviously you can't test it.

    Also, his notice about AVG appears to be outdated, since AVG Anti-Malware doesn't even appear on his test anymore, and is replaced by the new AVG 8.

    AFAIK that's exactly how AV-C selects their sample sets as well - by running the samples through scanners. AV-Test doesn't publish their methodology to the best of my knowledge. So I think it's still a stretch to label the two differently.
     
  15. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    8,183
    Location:
    The land of no identity :D
    The thing is that he has used AVG 8 Free, which does indeed have the same on-demand detection rate (apart from that of rootkits) as AVG 8 Pro. The mention of AVG Anti-Malware is a leftover from past tests......Thing is that while writing the specifics of the test, he makes a few mistakes but usually he fixes them after he is notified of the error.
     
  16. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,856
    Location:
    Innsbruck (Austria)
    NO -AV-Comparatives DOES NOT DO THAT.
    And AV-Test has also a published methodology on their website afair.
     
  17. trjam

    trjam Registered Member

    Joined:
    Aug 18, 2006
    Posts:
    9,102
    Location:
    North Carolina USA
    when I look at his site, it looks as legit as the rest. I think based on his test bed his results are fairly consistent with what you may find.:cool:
     
  18. C.S.J

    C.S.J Massive Poster

    Joined:
    Oct 16, 2006
    Posts:
    5,029
    Location:
    this forum is biased!
    am i right in thinking they use existing av's to calculate if its malware or not...

    if so, this test cannot be regarded as valid in any sense. :rolleyes:
     
  19. trjam

    trjam Registered Member

    Joined:
    Aug 18, 2006
    Posts:
    9,102
    Location:
    North Carolina USA
  20. jrmhng

    jrmhng Registered Member

    Joined:
    Nov 4, 2007
    Posts:
    1,268
    Location:
    Australia
    If he makes many mistakes in publishing and doesn't fix it up until someone notifies him, what does that tell us about how he conducts tests.

    Which other malware tester just says, oh ops the program doesn't work, lets drop it from our tests

    In anycase, I feel bad making all these points against him. Good on him for giving it a go. I just thought it was wierd that no one questioned his methodology at the start.

    Cheers
    Jeremy
     
  21. tiagozt

    tiagozt Registered Member

    Joined:
    Feb 28, 2004
    Posts:
    331
    Norton and Mcafee never were TOP AVs in detection.... They only get TOP rating in marketing...
     
  22. The Hammer

    The Hammer Registered Member

    Joined:
    May 12, 2005
    Posts:
    5,716
    Location:
    Toronto Canada
    Global warming. I can feel the hot air.:rolleyes:
     
  23. lodore

    lodore Registered Member

    Joined:
    Jun 22, 2006
    Posts:
    9,064
    i agree
     
  24. jrmhng

    jrmhng Registered Member

    Joined:
    Nov 4, 2007
    Posts:
    1,268
    Location:
    Australia
    Hmm I wonder how they market and sell themselves to enterprise that make them market leaders.
     
  25. Someone

    Someone Registered Member

    Joined:
    Jan 18, 2008
    Posts:
    1,106
    I think its mainly because they have been around for a long time.
     
Loading...
Thread Status:
Not open for further replies.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.