Big differences between 2 trustworthy tests ?

Discussion in 'other anti-virus software' started by aluckystar, May 4, 2007.

Thread Status:
Not open for further replies.
  1. aluckystar
    Offline

    aluckystar Registered Member

    According to the recent report by AV-comparatives, Tests done by AV-Test and AV-comparatives are trustworthy.

    But recently I found something interesting.
    Let's see.

    AV-comparatives did their On-demand comparative (as usual) on February, 2007. In the test, "Rising Antivirus"(acturally take part in 2nd group test) had a detection percentage of 69% for backdoors/trojans and 71% for Total.(Did not get the standard certification level)


    AV-Test did 2 tests for PC-WELT(a German IT Magazine) on September 2006 and November 2006.
    (They test 290,000 different files with Trojan horses from the year 2006. Viruses and worms do not participate in this test. In the meantime Trojan horses constitute such as Bots, Backdoors and Spyware more than 80 per cent of the mark commodity in the circulation.)

    In these 2 tests "Rising Antivirus" got 91.18% and 90.11%. In the test done in Sep 2006, it performs even better than Panda(90.45%), Dr Web(90.38%), Trend Micro(90.03%), Ikarus(84,77%) and VBA32(81,28%).

    And AV-Test did a test for PC WORLD (an USA IT Magazine) on April, 2007.. The samples contains all kind of viruses.

    In this test, the result is close to the test done by AV-Comparatives on Feb, 2007. See below(I listed all the repeated products in 2 tests), The result is very close :



    My Question is : Why the test result is so different on some product (especially on Rising Antivirus) ?

    The same phenomenon happen on F-Prot, it had a high detection rate of Backdoors and Trojans(92%, Feb 2007) in AV-Comparatives' Test and a low detection rate for Backdoors and Trajans(78%, April 2007) in AV-Test.


    You may say that they use different samples when they test.

    But by contrast, most of the other softwares have close results in these two tests.
    (I checked the results of two tests(Test AV-Comparatives did in Feb 2007 and Test AV-Test did in Nov 2006), and I found the detect percentage for Backdoor/Trojans of Kaspersky, Symantec Norton, AntiVir, AVK, Bitdefender, Dr.Web, Fortinet, Mcafee, Norman, NOD32 are all very close in two tests. )

    So, Can somebody tell me why ?
    Last edited: May 4, 2007
  2. Firecat
    Offline

    Firecat Registered Member

    Re: Big differences between 2 turstworthy tests ?

    1) Regarding F-Prot, AV-test used F-Prot 3.x instead of 6.x for the testing, so that explains the somewhat low results :)

    2) There are some very nice differences between AV-comparatives' test sets and AV-test.org's test sets. Probably Rising did not do well on detecting older malware on Clementi's test set, while at the same time AV-test uses somewhat newer samples. :)

    Just a speculation though. A wide variety of tests show a wide range of results due to many reasons. One can't go and determine the how and why of it unless he/she has access to the sample set.
  3. IBK
    Offline

    IBK AV Expert

    Re: Big differences between 2 turstworthy tests ?

    _maybe_ because av-comparatives does not include spywares, clients, tools, etc. in its trojan/backdoor sets.
  4. Firecat
    Offline

    Firecat Registered Member

    Re: Big differences between 2 turstworthy tests ?

    Spywares are counted differently from trojans and backdoors I think for AV-test....
  5. aluckystar
    Offline

    aluckystar Registered Member

    Re: Big differences between 2 turstworthy tests ?

    It is interesting that

    a product (like Rising Antivirus) that

    even did not get the "STANDARD CERTIFICATION LEVEL" (AV-Comparatives' Result)

    BEAT (AV-Test's Result)

    the product (like Dr.web, F-prot) that

    often get "ADVANCED CERTIFICATION LEVEL".(AV-Comparatives' Result)


    And these two tests are both Trustworthy according to the words by one of these two test orgnizations. :D
  6. Firecat
    Offline

    Firecat Registered Member

    Re: Big differences between 2 turstworthy tests ?

    F-Prot 3.x - STANDARD certification AV. F-Prot 4.x - ADVANCED certification.
  7. aluckystar
    Offline

    aluckystar Registered Member

    Re: Big differences between 2 turstworthy tests ?

    Results are complete opposite.
    Did F-Prot make a great progress or Rising did not "rising" any more ?
  8. TonyW
    Offline

    TonyW Registered Member

    Re: Big differences between 2 turstworthy tests ?

    The differences between the two tests are probably attrituable to samples used in the tests, versions of products being used as in the case of F-Prot or what level the program was tested at i.e. at default or higher setting. Or a combination of those.
  9. ErikAlbert
    Offline

    ErikAlbert Registered Member

    Re: Big differences between 2 turstworthy tests ?

    Different test beds = different results = other winner = other losers.
  10. Firecat
    Offline

    Firecat Registered Member

    Re: Big differences between 2 turstworthy tests ?

    But I will admit Rising's results are VERY strange because it breaks any consistency there was between the various AVs on both tests.
  11. halcyon
    Offline

    halcyon Registered Member

    Re: Big differences between 2 turstworthy tests ?

    aluckystar has a good point.

    IF the results are down to mostly difference in testing methodologies, THEN:

    0) Most people are not AV experts and will only look at the final ranking (truly understanding the raw results takes expertise in the field, something which 99.9999999% people lack)

    1) Only programs that do well in both tests should be even considered for recommendation for an average user

    2) Is there a certain other test setup, which is equally 'good' and which would produce a third differeing set of results?

    To summarize: how can a non-expert choose a fw if the experts opinions can differ so wildly?
  12. MalwareDie
    Offline

    MalwareDie Registered Member

    Re: Big differences between 2 turstworthy tests ?

  13. Firecat
    Offline

    Firecat Registered Member

    Re: Big differences between 2 turstworthy tests ?

  14. Firefighter
    Offline

    Firefighter Registered Member

    Re: Big differences between 2 turstworthy tests ?

    Or maybe just because Rising hasn't got the former "missed" samples from Av-Comparatives to increase the total detection with old samples (how could it, when it has been tested only the first time?). On the other hand, maybe the samples in Av-Test.org are just newer! :doubt:

    Best regards,
    Firefighter!
  15. MalwareDie
    Offline

    MalwareDie Registered Member

    none of us can really prove anything at all. Maybe we should get the opinion of some vendors. We could could ask IC, Marcos, Stefan, vlk, Serge Popov and see what they think. I think most of them wil lean toward av-comparatives.
  16. Firefighter
    Offline

    Firefighter Registered Member

    Somehow I just believe that even you does not believe what you said!

    In my mind, two different tests, two different point of views, that's it. Actually, none of these reflects the everyday protection level, which is much worse with all solutions, unfortunately! :cool:

    Best regards,
    Firefighter!
  17. JerryM
    Offline

    JerryM Registered Member

    Hi Firefighter,
    Why should protectiion level be "much worse with all solutions?"

    Best,
    Jerry
  18. Firecat
    Offline

    Firecat Registered Member

    That is your opinion. Personally I do not think anyone is going to take sides on a "which test is better" dispute. The fact is both are very well regarded in the industry, and I cannot think of anyone who'd have anything bad to say about both these organizations (apart from minor suggestions for improvement of course :D). If there was some complaint with the methodology of either AV-test or AV-comparatives, there are enough AV experts here who would have voiced there complaint. The only company I know of that would possibly not like AV-test.org is Eset, but those disputes were a long time ago and I do not think they have a lasting impact now. :)
  19. Firecat
    Offline

    Firecat Registered Member

    Because when you consider the "zero-day" threats, only those AVs with good heuristics or those AVs which have frequent updates are able to protect you well. Otherwise the signature is added too late. Or at least I think thats what Firefighter is trying to say.

    IMO real-world protection is not so bad at all if you have a reasonably good AV. :)
  20. MalwareDie
    Offline

    MalwareDie Registered Member

    Gah I am just a depserate person. you lean towards av-test saying that they hav more resources than av-compratives. and that is why i keep badgering you because mroe resources doesnt mean it is better. You said that Av-comparatives uess older sampels than av-test which you dont nkow for sure since neither av-test or av-comparatives sends you their samples. And IBK didnt seem to agree with you when you said that.
  21. Firecat
    Offline

    Firecat Registered Member

    MalwareDie,

    You seem to have misinterpreted me slightly :)

    More resources doesn't mean its worse either right? ;)

    I said that AV-comparatives uses many older samples, the intention was to say that both AV-test and AV-comparatives use older samples, and not to say that AV-comparatives' samples are older than AV-test or vice versa. :)

    Yeah, but if you noticed, I had added the comments "maybe" and "probably" while making statements in that direction, which means it is not a confirmed thing. :)
  22. MalwareDie
    Offline

    MalwareDie Registered Member

    Okay il stop bothering you now.
  23. veri
    Offline

    veri Registered Member

    Why is Antivir absent from the PC World ranking (here) but comparable as said above? How's that work?

    I'm thinking of either going back to Nod32, Antivir, or Symantec Corporate (from ESS), so I'm rather interested.
  24. The Hammer
    Offline

    The Hammer Registered Member

    You mean he might have possibly but not definitively gotten things indirectly incorrect when interpreting the general thrust of what you may have been trying to say. Without allowing yourself to be positively pinned down to a absolute position . ;)
    Last edited: May 4, 2007
  25. The Hammer
    Offline

    The Hammer Registered Member

    The tested software were all Vista compatible. Did Antivir have a Vista version at the time of the test?
Thread Status:
Not open for further replies.