New AV-comparatives' report on AV testing sites

Discussion in 'other anti-virus software' started by Firecat, Apr 21, 2007.

Thread Status:
Not open for further replies.
  1. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    8,251
    Location:
    The land of no identity :D
    Well, AV-comparatives has made a new report on AV testing sites, which list the trustworthy and untrustworthy testing sites, as well as gives some valuable information on what to look for when looking at an AV-test. :)

    You can find it on the lower portion on the Comparatives page, at the same area where the single product test reports are located.

    I have a few thoughts on reading this article. Firstly, there are a few things I've learned from contacting virus.gr and malware-test.com (at least they are open to discussion ;) )

    1) Malware-Test is revising its methodology at the moment. In the future they will ensure that all products are updated on the same day. There are also a few other changes planned, but these are not known to me. I'll try to find out more. :)

    2) virus.gr is a rather strange thing. It is true that most VX collections seem to favour a particular scanner (no need to name it now). All I will say is that Antony Petrakis is open to suggestions and I have suggested to him some things, which I think he will follow in the future. While this should make his tests slightly more practical in the future, there are still a few more (major) flaws which I think will need to be looked at. Either way, Antony is a very busy man, and due to that communication has been slow, thus making this somewhat difficult. :(

    One question I had is how did IBK find out that malware-test is counting detections wrongly? I mean, till now I thought that Malware-Test was flawed because of multiple corrupted samples due to honeypot based testing and also because the tester did not update all the products on the same day.
     
    Last edited: Apr 21, 2007
  2. aigle

    aigle Registered Member

    Joined:
    Dec 14, 2005
    Posts:
    11,164
    Location:
    UK / Pakistan
    OT/ malware-test is under DDOS now!!
     
  3. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    8,251
    Location:
    The land of no identity :D
    Regarding "Tests run by users and/or unexperienced peoples": I do not speak for everyone when I speak on this issue, but I do agree that such tests made by users on forums may not be truly reliable due non-analysed and possibly corrupt samples and also due to small sample size. Luckily enough, one of the vendors I send the files for analysis to gives me a detailed description of which files are junk, which are corrupt and which are not really malware. So in this sense I'm somewhat lucky, and I thank this vendor with all my heart for providing this information. :)
     
  4. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    8,251
    Location:
    The land of no identity :D
    :eek::eek::eek::eek::eek::eek::eek:
     
  5. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    8,251
    Location:
    The land of no identity :D
    One more thing that came to my head is that one testing authority is not even mentioned in this report - scheinsicherheit. I was not very attuned with the events when ntl used to go on about, commenting about AVs and their weaknesses, and sometimes posting test results on his forum. I would appreciate some info on what this was and whether it was (is) reliable or not....:doubt:
     
  6. FRug

    FRug Registered Member

    Joined:
    Feb 7, 2006
    Posts:
    309
    Scheinsicherheit is a site run by folks who keep yapping about the dangers of patched and repacked malware, as if AV companies wouldn't know or wouldn't care about the danger.
    Their testbed consists of such manually repacked and self-patched malware which is a debatable practice at best. Their forum has been hacked multiple times and the response usually was along the lines of "so what?". Yup. Really professional attitude. Anyway, there haven't been any tests by them for quite a while, and they certainly can't be counted as a "testing institution". If i'd count the site as anything then as an inquisition advocator for burning the oh so clueless AV companies of this world at the stake, who keep ignoring the oh so tragic 'new trend' of repacking and patching just because they are lazy bastards.
    I've followed quite a few discussions in several forums involving Nautilus who seems to be the guy running the site, and am positive that he's not someone worth listening to on the topic of security since he has a severe attitude problem.

    Nuff said, don't waste your time there.
     
  7. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    8,251
    Location:
    The land of no identity :D
    Thanks for the clarification, it is highly appreciated. :)
     
  8. aluckystar

    aluckystar Registered Member

    Joined:
    May 30, 2006
    Posts:
    66
    Location:
    Paris of the East
    Malware-Test Lab was founded by a former engineer from TrendMicro.:rolleyes:
     
  9. extratime

    extratime Registered Member

    Joined:
    Oct 14, 2005
    Posts:
    100
    Ok got done reading the report. I think it is an excellent report. Some of the AV testing sites may not like it but hopefully this will spur them to improve their testing.

    Good job IBK!
     
  10. bellgamin

    bellgamin Registered Member

    Joined:
    Aug 1, 2002
    Posts:
    8,102
    Location:
    Hawaii
    If ANYone but AV-comp did this, I would have to say: "A test site that rates it competitors? Hmmm..."
     
  11. C.S.J

    C.S.J Massive Poster

    Joined:
    Oct 16, 2006
    Posts:
    5,029
    Location:
    this forum is biased!
    i read it,

    sounds like a 'we're right, your wrong' report and reads like a childrens playground battle for status :D

    not sure why av-comp have brought a pdf like this.
     
  12. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,886
    Location:
    Innsbruck (Austria)
    then you misread it. it states the importance of the various av tests and certifications and that one test alone is meaningless.
    it also summarizes the faults of the non trustworthy tests, things that you can find everywhere discussed in various forums.
     
  13. EliteKiller

    EliteKiller Registered Member

    Joined:
    Jan 18, 2007
    Posts:
    1,138
    Location:
    TX
    IBK, why not mention that users should not put too much faith in a VB100 or other "marketing" based certification tests since even mediocre AV's can pass them, but fail to offer adequate protection beyond the scope of the test set?
     
  14. lucas1985

    lucas1985 Retired Moderator

    Joined:
    Nov 9, 2006
    Posts:
    4,047
    Location:
    France, May 1968
    Because the article focuses on accuracy of test methodologies, not on the usefulness of such tests. VB100 tests are accurate (for the most part), but they don't say much about real life detection.
     
  15. Firecat

    Firecat Registered Member

    Joined:
    Jan 2, 2005
    Posts:
    8,251
    Location:
    The land of no identity :D
    How about the fact that VB100 (or Check Mark or anything else) is a larger organization than AV-comparatives? So if a legal battle came out (on the charges of so-called defamation), then it would be catastrophic :ninja:

    <The above statement is just the cynic side of me :p>

    Anyway, as lucas1985 said, the aim of the article is to focus on the accuracy and reliability of such tests, rather than the implications or usefulness of it. :)
     
  16. EliteKiller

    EliteKiller Registered Member

    Joined:
    Jan 18, 2007
    Posts:
    1,138
    Location:
    TX
    Accuracy and usefulness should go hand in hand.

    The title of the article is "An overview on which testing sites can be trusted and which can not", not which testing methodologies. The article is based on IBK's opinion and expertise in the field. I also see a lot of opinions in the article so I don't see the problem in adding yet another opinion on how you should/should not take some of the well-known organizations results with a grain of salt. TTBOMK there's no defamation when you back up statements with facts.
     
  17. lucas1985

    lucas1985 Retired Moderator

    Joined:
    Nov 9, 2006
    Posts:
    4,047
    Location:
    France, May 1968
    IMHO, "An overview on which testing sites can be trusted and which can not" means that organization X uses a reputable/proven/unbiased/verifiable methodology and you can trust their results. Whatever you do with those results is another thing.
     
  18. C.S.J

    C.S.J Massive Poster

    Joined:
    Oct 16, 2006
    Posts:
    5,029
    Location:
    this forum is biased!
    personally i look at ALL testing sites and get an average,

    sure some testers will say X is bad while Y is good, but to check them all and get an average, seems a good way to get an overall opinion of the program.

    then go into your own tryouts of the software, if it agrees with your system and is doing its job and you like it, purchase :)
     
Thread Status:
Not open for further replies.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.