Big differences between 2 trustworthy tests ?

[aluckystar]

According to the recent report by AV-comparatives, Tests done by AV-Test and AV-comparatives are trustworthy.

But recently I found something interesting.
Let's see.

AV-comparatives did their On-demand comparative (as usual) on February, 2007. In the test, "Rising Antivirus"(acturally take part in 2nd group test) had a detection percentage of 69% for backdoors/trojans and 71% for Total.(Did not get the standard certification level)


AV-Test did 2 tests for PC-WELT(a German IT Magazine) on September 2006 and November 2006.
（They test 290,000 different files with Trojan horses from the year 2006. Viruses and worms do not participate in this test. In the meantime Trojan horses constitute such as Bots, Backdoors and Spyware more than 80 per cent of the mark commodity in the circulation.)

In these 2 tests "Rising Antivirus" got 91.18% and 90.11%. In the test done in Sep 2006, it performs even better than Panda(90.45%), Dr Web(90.38%), Trend Micro(90.03%), Ikarus(84,77%) and VBA32(81,28%).

And AV-Test did a test for PC WORLD (an USA IT Magazine) on April, 2007.. The samples contains all kind of viruses.

In this test, the result is close to the test done by AV-Comparatives on Feb, 2007. See below(I listed all the repeated products in 2 tests), The result is very close :

Kaspersky 6.0 AV-Comparatives:97.89% AV-Test: 96%
Norton 2007 AV-Comparatives:96.83% AV-Test: 96%
Bitdefender v10 AV-Comparatives:96.11% AV-Test: 96%
ESET NOD32 AV-Comparatives:96.71% AV-Test: 90%
Avast! AV-Comparatives:93.86% AV-Test: 92%
AVG AV-Comparatives:96.37% AV-Test: 91%
Trend Micro AV-Comparatives:87% AV-Test: 82%
(AV-Comparatives tested Trend Micro PC-Cillin on April because it is a single product test. So Trend Micro had 2 month more to collect samples, the result 87% maybe overestimated )

My Question is : Why the test result is so different on some product (especially on Rising Antivirus) ?

The same phenomenon happen on F-Prot, it had a high detection rate of Backdoors and Trojans(92%, Feb 2007) in AV-Comparatives' Test and a low detection rate for Backdoors and Trajans(78%, April 2007) in AV-Test.


You may say that they use different samples when they test.

But by contrast, most of the other softwares have close results in these two tests.
(I checked the results of two tests(Test AV-Comparatives did in Feb 2007 and Test AV-Test did in Nov 2006), and I found the detect percentage for Backdoor/Trojans of Kaspersky, Symantec Norton, AntiVir, AVK, Bitdefender, Dr.Web, Fortinet, Mcafee, Norman, NOD32 are all very close in two tests. )

So, Can somebody tell me why ?

[Firecat]

1) Regarding F-Prot, AV-test used F-Prot 3.x instead of 6.x for the testing, so that explains the somewhat low results

2) There are some very nice differences between AV-comparatives' test sets and AV-test.org's test sets. Probably Rising did not do well on detecting older malware on Clementi's test set, while at the same time AV-test uses somewhat newer samples.

Just a speculation though. A wide variety of tests show a wide range of results due to many reasons. One can't go and determine the how and why of it unless he/she has access to the sample set.

[IBK]

_maybe_ because av-comparatives does not include spywares, clients, tools, etc. in its trojan/backdoor sets.

[Firecat]

Originally Posted by IBK

_maybe_ because av-comparatives does not include spywares, clients, tools, etc. in its trojan/backdoor sets.

Spywares are counted differently from trojans and backdoors I think for AV-test....

[aluckystar]

It is interesting that

a product (like Rising Antivirus) that

even did not get the "STANDARD CERTIFICATION LEVEL" (AV-Comparatives' Result)

BEAT (AV-Test's Result)

the product (like Dr.web, F-prot) that

often get "ADVANCED CERTIFICATION LEVEL".(AV-Comparatives' Result)


And these two tests are both Trustworthy according to the words by one of these two test orgnizations.

[Firecat]

F-Prot 3.x - STANDARD certification AV. F-Prot 4.x - ADVANCED certification.

[aluckystar]

AV-Comparatives February, 2007:
Rising 69% (Backdoors/Trojans) 71%(Total)
F-Prot 92% (Backdoors/Trojans) 94%(Total)


AV-Test (Trojans/Backdoors, Bots) :
Rising 91%(Sep, 2006) 90% (Nov. 2006)
F-Prot 78%(Sep, 2006) 81% (Nov. 2006)

Results are complete opposite.
Did F-Prot make a great progress or Rising did not "rising" any more ?

[TonyW]

The differences between the two tests are probably attrituable to samples used in the tests, versions of products being used as in the case of F-Prot or what level the program was tested at i.e. at default or higher setting. Or a combination of those.

[ErikAlbert]

Different test beds = different results = other winner = other losers.

[Firecat]

Originally Posted by ErikAlbert

Different test beds = different results = other winner = other losers.

But I will admit Rising's results are VERY strange because it breaks any consistency there was between the various AVs on both tests.

[halcyon]

aluckystar has a good point.

IF the results are down to mostly difference in testing methodologies, THEN:

0) Most people are not AV experts and will only look at the final ranking (truly understanding the raw results takes expertise in the field, something which 99.9999999% people lack)

1) Only programs that do well in both tests should be even considered for recommendation for an average user

2) Is there a certain other test setup, which is equally 'good' and which would produce a third differeing set of results?

To summarize: how can a non-expert choose a fw if the experts opinions can differ so wildly?

[MalwareDie]

[quote=Firecat]
2) There are some very nice differences between AV-comparatives' test sets and AV-test.org's test sets. Probably Rising did not do well on detecting older malware on Clementi's test set, while at the same time AV-test uses somewhat newer samples.

QUOTE]


Av-test is the one that uses older malware. You can tell by looking at the numbers. Between their september and november test they used more than 500 000 trojans. there are not that many trojans released in one year. So they are the ones using/reusing old samples or even corryuped samples. Just because Av-test is larger and has more resources does not mean they do a better job at getting new malware and providing better results. the fact that rising does so well on av-test makes av-test a bs organization.

[Firecat]

[quote=MalwareDie]

Originally Posted by Firecat

2) There are some very nice differences between AV-comparatives' test sets and AV-test.org's test sets. Probably Rising did not do well on detecting older malware on Clementi's test set, while at the same time AV-test uses somewhat newer samples.

QUOTE]


Av-test is the one that uses older malware. You can tell by looking at the numbers. Between their september and november test they used more than 500 000 trojans. there are not that many trojans released in one year. So they are the ones using/reusing old samples or even corryuped samples. Just because Av-test is larger and has more resources does not mean they do a better job at getting new malware and providing better results. the fact that rising does so well on av-test makes av-test a bs organization.

Not really. The fact is that AV-comparatives also uses many older samples. AV-test has always been around a lot more than AV-comparatives have been, so it can also be safe to say AV-test has more established sources for getting malware samples.

Between their september and november test they used more than 500 000 trojans

Wrong, this misconception is because in September the 290,000 trojans included backdoors, bots and zombies, which were given separate categories in the November test. If we count just the bots, backdoors, zombies and trojans together for November the total becomes 383000.

there are not that many trojans released in one year

NOBODY has an exact estimate of how many malware samples are released into the Net per year....

Just because Av-test is larger and has more resources does not mean they do a better job at getting new malware and providing better results.

I could say a similar statement for even VB100 or any other test organization. This does not mean anything at all. The industry respects AV-test, why don't you? Because AVG (without Ewido engine) and Rising perform better than you expect, and NOD32 performs slightly lower than expected? I have seen a lot of surprise recently about AVG's detection rates, many people simply cannot come to terms with the fact that it has improved. Symantec saga has been rewritten....

the fact that rising does so well on av-test makes av-test a bs organization.

Experts across the world do not share your opinion on this. AV-Test is as reliable a testing organization as anyone will ever get. Their trojan database is much more expansive than AV-comparatives, and that is mostly why people are seeing such different results. Now, where and how people get their samples is another matter altogether, but AV-test has always favoured those AVs with good trojan detection rate, and their polymorphic tests are consisted of several polymorphic viruses rather than some 10-12 which are intended to provide only a "demo" and inform users about flexibility of the engine. Of course, it is possible that the method of testing polymorphic virus detection is different in AV-test and AV-comparatives, so maybe there cannot be a direct comparison between the two tests in this regard.

I only made that comment about Rising not detecting older samples on AV-comparatives because I had asked this earlier and I was given a "maybe" reply from IBK. So it may or may not be true, at this moment one cannot be really sure. One thing to be noted is that Rising lacks in sources of obtaining European malware. If there is any difference in where the samples are obtained from between AV-test and AV-comparatives, that would explain Rising's detection rates.

[Firefighter]

Originally Posted by Firecat

But I will admit Rising's results are VERY strange because it breaks any consistency there was between the various AVs on both tests.

Or maybe just because Rising hasn't got the former "missed" samples from Av-Comparatives to increase the total detection with old samples (how could it, when it has been tested only the first time?). On the other hand, maybe the samples in Av-Test.org are just newer!

Best regards,
Firefighter!

[MalwareDie]

none of us can really prove anything at all. Maybe we should get the opinion of some vendors. We could could ask IC, Marcos, Stefan, vlk, Serge Popov and see what they think. I think most of them wil lean toward av-comparatives.

[Firefighter]

Originally Posted by MalwareDie

none of us can really prove anything at all...I think most of them wil lean toward av-comparatives.

Somehow I just believe that even you does not believe what you said!

In my mind, two different tests, two different point of views, that's it. Actually, none of these reflects the everyday protection level, which is much worse with all solutions, unfortunately!

Best regards,
Firefighter!

[JerryM]

Hi Firefighter,
Why should protectiion level be "much worse with all solutions?"

Best,
Jerry

[Firecat]

Originally Posted by MalwareDie

none of us can really prove anything at all. Maybe we should get the opinion of some vendors. We could could ask IC, Marcos, Stefan, vlk, Serge Popov and see what they think. I think most of them wil lean toward av-comparatives.

That is your opinion. Personally I do not think anyone is going to take sides on a "which test is better" dispute. The fact is both are very well regarded in the industry, and I cannot think of anyone who'd have anything bad to say about both these organizations (apart from minor suggestions for improvement of course ). If there was some complaint with the methodology of either AV-test or AV-comparatives, there are enough AV experts here who would have voiced there complaint. The only company I know of that would possibly not like AV-test.org is Eset, but those disputes were a long time ago and I do not think they have a lasting impact now.

[Firecat]

Originally Posted by JerryM

Hi Firefighter,
Why should protectiion level be "much worse with all solutions?"

Best,
Jerry

Because when you consider the "zero-day" threats, only those AVs with good heuristics or those AVs which have frequent updates are able to protect you well. Otherwise the signature is added too late. Or at least I think thats what Firefighter is trying to say.

IMO real-world protection is not so bad at all if you have a reasonably good AV.

[MalwareDie]

Gah I am just a depserate person. you lean towards av-test saying that they hav more resources than av-compratives. and that is why i keep badgering you because mroe resources doesnt mean it is better. You said that Av-comparatives uess older sampels than av-test which you dont nkow for sure since neither av-test or av-comparatives sends you their samples. And IBK didnt seem to agree with you when you said that.

[Firecat]

Originally Posted by MalwareDie

Gah I am just a depserate person. you lean towards av-test saying that they hav more resources than av-compratives. and that is why i keep badgering you because mroe resources doesnt mean it is better. You said that Av-comparatives uess older sampels than av-test which you dont nkow for sure since neither av-test or av-comparatives sends you their samples. And IBK didnt seem to agree with you when you said that.

MalwareDie,

You seem to have misinterpreted me slightly

mroe resources doesnt mean it is better.

More resources doesn't mean its worse either right?

You said that Av-comparatives uess older sampels than av-test which you dont nkow for sure since neither av-test or av-comparatives sends you their samples.

I said that AV-comparatives uses many older samples, the intention was to say that both AV-test and AV-comparatives use older samples, and not to say that AV-comparatives' samples are older than AV-test or vice versa.

And IBK didnt seem to agree with you when you said that.

Yeah, but if you noticed, I had added the comments "maybe" and "probably" while making statements in that direction, which means it is not a confirmed thing.

[MalwareDie]

Okay il stop bothering you now.

[veri]

Originally Posted by aluckystar

According to the recent report by AV-comparatives, Tests done by AV-Test and AV-comparatives are trustworthy.
...

But by contrast, most of the other softwares have close results in these two tests.
(I checked the results of two tests(Test AV-Comparatives did in Feb 2007 and Test AV-Test did in Nov 2006), and I found the detect percentage for Backdoor/Trojans of Kaspersky, Symantec Norton, AntiVir, AVK, Bitdefender, Dr.Web, Fortinet, Mcafee, Norman, NOD32 are all very close in two tests. )

So, Can somebody tell me why ?

Why is Antivir absent from the PC World ranking (here) but comparable as said above? How's that work?

I'm thinking of either going back to Nod32, Antivir, or Symantec Corporate (from ESS), so I'm rather interested.

[The Hammer]

Originally Posted by Firecat

MalwareDie,

You seem to have misinterpreted me slightly






Yeah, but if you noticed, I had added the comments "maybe" and "probably" while making statements in that direction, which means it is not a confirmed thing.

You mean he might have possibly but not definitively gotten things indirectly incorrect when interpreting the general thrust of what you may have been trying to say. Without allowing yourself to be positively pinned down to a absolute position .

[The Hammer]

Originally Posted by veri

Why is Antivir absent from the PC World ranking (here) but comparable as said above? How's that work?

I'm thinking of either going back to Nod32, Antivir, or Symantec Corporate (from ESS), so I'm rather interested.

The tested software were all Vista compatible. Did Antivir have a Vista version at the time of the test?