Big differences between 2 trustworthy tests ?

According to the recent report by AV-comparatives, Tests done by AV-Test and AV-comparatives are trustworthy.

But recently I found something interesting.
Let's see.

AV-comparatives did their On-demand comparative (as usual) on February, 2007. In the test, "Rising Antivirus"(acturally take part in 2nd group test) had a detection percentage of 69% for backdoors/trojans and 71% for Total.(Did not get the standard certification level)


AV-Test did 2 tests for PC-WELT(a German IT Magazine) on September 2006 and November 2006.
（They test 290,000 different files with Trojan horses from the year 2006. Viruses and worms do not participate in this test. In the meantime Trojan horses constitute such as Bots, Backdoors and Spyware more than 80 per cent of the mark commodity in the circulation.)

In these 2 tests "Rising Antivirus" got 91.18% and 90.11%. In the test done in Sep 2006, it performs even better than Panda(90.45%), Dr Web(90.38%), Trend Micro(90.03%), Ikarus(84,77%) and VBA32(81,28%).

And AV-Test did a test for PC WORLD (an USA IT Magazine) on April, 2007.. The samples contains all kind of viruses.

In this test, the result is close to the test done by AV-Comparatives on Feb, 2007. See below(I listed all the repeated products in 2 tests), The result is very close :

My Question is : Why the test result is so different on some product (especially on Rising Antivirus) ?

The same phenomenon happen on F-Prot, it had a high detection rate of Backdoors and Trojans(92%, Feb 2007) in AV-Comparatives' Test and a low detection rate for Backdoors and Trajans(78%, April 2007) in AV-Test.


You may say that they use different samples when they test.

But by contrast, most of the other softwares have close results in these two tests.
(I checked the results of two tests(Test AV-Comparatives did in Feb 2007 and Test AV-Test did in Nov 2006), and I found the detect percentage for Backdoor/Trojans of Kaspersky, Symantec Norton, AntiVir, AVK, Bitdefender, Dr.Web, Fortinet, Mcafee, Norman, NOD32 are all very close in two tests. )

So, Can somebody tell me why ?

 

Re: Big differences between 2 turstworthy tests ?

1) Regarding F-Prot, AV-test used F-Prot 3.x instead of 6.x for the testing, so that explains the somewhat low results

2) There are some very nice differences between AV-comparatives' test sets and AV-test.org's test sets. Probably Rising did not do well on detecting older malware on Clementi's test set, while at the same time AV-test uses somewhat newer samples.

Just a speculation though. A wide variety of tests show a wide range of results due to many reasons. One can't go and determine the how and why of it unless he/she has access to the sample set.

 

Re: Big differences between 2 turstworthy tests ?

_maybe_ because av-comparatives does not include spywares, clients, tools, etc. in its trojan/backdoor sets.

 

Re: Big differences between 2 turstworthy tests ?

Spywares are counted differently from trojans and backdoors I think for AV-test....

 

Re: Big differences between 2 turstworthy tests ?

It is interesting that

a product (like Rising Antivirus) that

even did not get the "STANDARD CERTIFICATION LEVEL" (AV-Comparatives' Result)

BEAT (AV-Test's Result)

the product (like Dr.web, F-prot) that

often get "ADVANCED CERTIFICATION LEVEL".(AV-Comparatives' Result)


And these two tests are both Trustworthy according to the words by one of these two test orgnizations.

 

Re: Big differences between 2 turstworthy tests ?

F-Prot 3.x - STANDARD certification AV. F-Prot 4.x - ADVANCED certification.

 

Re: Big differences between 2 turstworthy tests ?

Results are complete opposite.
Did F-Prot make a great progress or Rising did not "rising" any more ?

 

Re: Big differences between 2 turstworthy tests ?

The differences between the two tests are probably attrituable to samples used in the tests, versions of products being used as in the case of F-Prot or what level the program was tested at i.e. at default or higher setting. Or a combination of those.

 

Re: Big differences between 2 turstworthy tests ?

Different test beds = different results = other winner = other losers.

 

Re: Big differences between 2 turstworthy tests ?

But I will admit Rising's results are VERY strange because it breaks any consistency there was between the various AVs on both tests.

 

Re: Big differences between 2 turstworthy tests ?

aluckystar has a good point.

IF the results are down to mostly difference in testing methodologies, THEN:

0) Most people are not AV experts and will only look at the final ranking (truly understanding the raw results takes expertise in the field, something which 99.9999999% people lack)

1) Only programs that do well in both tests should be even considered for recommendation for an average user

2) Is there a certain other test setup, which is equally 'good' and which would produce a third differeing set of results?

To summarize: how can a non-expert choose a fw if the experts opinions can differ so wildly?

 

Re: Big differences between 2 turstworthy tests ?

 

Re: Big differences between 2 turstworthy tests ?

 

Re: Big differences between 2 turstworthy tests ?

Or maybe just because Rising hasn't got the former "missed" samples from Av-Comparatives to increase the total detection with old samples (how could it, when it has been tested only the first time?). On the other hand, maybe the samples in Av-Test.org are just newer!

Best regards,
Firefighter!

 

none of us can really prove anything at all. Maybe we should get the opinion of some vendors. We could could ask IC, Marcos, Stefan, vlk, Serge Popov and see what they think. I think most of them wil lean toward av-comparatives.

 

Somehow I just believe that even you does not believe what you said!

In my mind, two different tests, two different point of views, that's it. Actually, none of these reflects the everyday protection level, which is much worse with all solutions, unfortunately!

Best regards,
Firefighter!

 

Hi Firefighter,
Why should protectiion level be "much worse with all solutions?"

Best,
Jerry

 

That is your opinion. Personally I do not think anyone is going to take sides on a "which test is better" dispute. The fact is both are very well regarded in the industry, and I cannot think of anyone who'd have anything bad to say about both these organizations (apart from minor suggestions for improvement of course ). If there was some complaint with the methodology of either AV-test or AV-comparatives, there are enough AV experts here who would have voiced there complaint. The only company I know of that would possibly not like AV-test.org is Eset, but those disputes were a long time ago and I do not think they have a lasting impact now.

 

Because when you consider the "zero-day" threats, only those AVs with good heuristics or those AVs which have frequent updates are able to protect you well. Otherwise the signature is added too late. Or at least I think thats what Firefighter is trying to say.

IMO real-world protection is not so bad at all if you have a reasonably good AV.

 

Gah I am just a depserate person. you lean towards av-test saying that they hav more resources than av-compratives. and that is why i keep badgering you because mroe resources doesnt mean it is better. You said that Av-comparatives uess older sampels than av-test which you dont nkow for sure since neither av-test or av-comparatives sends you their samples. And IBK didnt seem to agree with you when you said that.

 

MalwareDie,

You seem to have misinterpreted me slightly

More resources doesn't mean its worse either right?

I said that AV-comparatives uses many older samples, the intention was to say that both AV-test and AV-comparatives use older samples, and not to say that AV-comparatives' samples are older than AV-test or vice versa.

Yeah, but if you noticed, I had added the comments "maybe" and "probably" while making statements in that direction, which means it is not a confirmed thing.

 

Okay il stop bothering you now.

 

Why is Antivir absent from the PC World ranking (here) but comparable as said above? How's that work?

I'm thinking of either going back to Nod32, Antivir, or Symantec Corporate (from ESS), so I'm rather interested.

 

You mean he might have possibly but not definitively gotten things indirectly incorrect when interpreting the general thrust of what you may have been trying to say. Without allowing yourself to be positively pinned down to a absolute position .

 

The tested software were all Vista compatible. Did Antivir have a Vista version at the time of the test?