new results from AV-Test.org (Q1/2008)

[Valentin_Pletzer]

Hi guys!

I just wanted to let you know that Andreas Marx was kind enough to provide me with his newest test results. He is currently in Bilbao, Spain at the Anti-Malware Task Force Meeting.

I published the results in Blog (in german) http://blog.chip.de/0-security-blog/...2008-20080122/

If you have any questions, please feel free to leave a comment beneath the blog-entry.

Greetings from Munich
Valentin

[huangker]

Interesting. Not too many surprises there. A few things I noted though,

1) Why does command do worse than fprot when they are using the same engine?
2) Clam is improving especially it is only signature based
3) Microsoft is also improving (though other tests have already shown it has improved a fair bit since onecare v1). Seems to be very strong signature detection but low heuristics.
4) Eset strong on heuristics but not as good on signature scanning (does that surprise anyone?)
5) Just find the fact that VET is on the top of the false positive list and bottom of the detection rate sadistically funny.

[Blackcat]

Originally Posted by huangker

Interesting. Not too many surprises there. A few things I noted though,

1) Why does command do worse than fprot when they are using the same engine?

They are not. CSAV is still using the old 3 engine, so is equivalent to F-Prot 3. In contrast FPAV 6 is tested here and as shown has a much higher detection rate.

[Sputnik]

Nice, thanks a lot for posting. Personally I'm very pleased to see the peformance of avast!, their huge signature additions are paying off. Also TrendMicro is in the detection elevator, best detection of the top 3 brands (Symantec, McAfee, TrendMicro)!

[trjam]

English Translation:

That is really a beautiful surprise. In my p.o. box nevertheless actually just the all-newest virus scanner test results arrived. The results come directly of Andreas's Marx and its test laboratory AV test those do not want I you naturally not to withhold. Security Suiten conditions 7 January 2008 under Windows XP SP2 (English) were tested. With all products it concerns the optimum version (not however the beta) The test categories read as follows: - signature-based test of 1 million Malware Samples from the last 6 months (thus no outdated viruses) - False positive test with 65,000 clean files - pro-active recognition with: + 3.500 samples in retrospective test (the signatures are not called one week updated and it looked which new Samples be still recognized now) + 20 active Samples for the behavior-based test - response times (based on 55 Samples in the year 2007) - root kit recognition (12 active Samples) First once the total valuation:

[dawgg]

I'm a little lost... can someone please inform me; If there are 1 million malware samples used, why do some AVs detect more than a million?

I'm surprised Avast done so well and Antivir had so few FPs.
Not surprised Antivir and Kaspersky have amongst the fastest response times.
Suprised with WebWasher getting only 2 FPs
Wouldnt have expected AntiVir to get + for Proactive Detection and F-Secure to get ++

[xandros]

good job avira antivir & avast

i read many things about antivir many sites and its excellent

[Stijnson]

4) Eset strong on heuristics but not as good on signature scanning (does that surprise anyone?)

I'm a bit technically challenged, so can someone explain what this means?

[Steel]

The results of NOD in all Categories frighten me much. Whats happens here ?

[C.S.J]

Only 2 fps from drweb?

Am I reading this right?

[Xenophobe]

Originally Posted by Stijnson

I'm a bit technically challenged, so can someone explain what this means?

Eset did a poor job of detecting threats with signatures (which are issued in daily updates) and good in heuristics, which is a method to detect possible viruses.

[Stijnson]

Originally Posted by Xenephobe

Eset did a poor job of detecting threats with signatures (which are issued in daily updates) and good in heuristics, which is a method to detect possible viruses.

Hmmm, okay. Thanks Xenephobe.

I also see Symantec in the list. Does anyone know which version has been tested (where can I find this)? Is this the same as a Corporate version?

[solcroft]

Originally Posted by Xenephobe

Eset did a poor job of detecting threats with signatures (which are issued in daily updates) and good in heuristics, which is a method to detect possible viruses.

Not good enough to help its overall detection score, unfortunately. Do you mean to say that the testers turned off Eset's heuristics for this test?

[Paul Wilders]

Some vital information is missing as far as I'm concerned: no info concerning the testbed used as for the signature test for example. Is plain adware included for example? Smart people can come up with more questions like that I' sure

All in all, personally I'd like to see far more info about the test conditions before jumping to a conclusion.

That said: for the moment although lacking needed info: congrats to the ones who did score very well.

regards,

paul

[Paul Wilders]

Originally Posted by solcroft

...Do you mean to say that the testers turned off Eset's heuristics for this test?

...and there's the first smart question Has been tested out-of-the box, has there been tested after tweaking?

Keep them coming those questions, ladies and gents!

[Stijnson]

What I find a bit strange is that NOD32 always scores lower in AV-Test.org tests compared to the AV Comparatives'...
I guess it's also a matter of how things are being tested. I do hope these AV-Test results will be expanded with version numbers of the specified products though. Those seem to be missing.

[Dieselman]

Doesnt make me feel good about spending $40 on NOD32. Should have kept Avast for free.

[Paul Wilders]

Originally Posted by Stijnson

What I find a bit strange is that NOD32 always scores lower in AV-Test.org tests compared to the AV Comparatives'...
I guess it's also a matter of how things are being tested.

Bolded part: Bingo! Plus: what sort of samples have been tested?

regards,

Paul

[Paul Wilders]

Originally Posted by Dieselman

Doesnt make me feel good about spending $40 on NOD32. Should have kept Avast for free.

One should not jump to conclusions without knowing all the needed facts. And this does not in particular goes for NOD32, but for all Antiviruses tested .

regards,

paul

[aigle]

Overall detection of NOD 32 is not good though it has very good heuristics.
They must add a lot of signatures like Avira and others.

Originally Posted by Paul Wilders

...and there's the first smart question Has been tested out-of-the box, has there been tested after tweaking?

Keep them coming those questions, ladies and gents!

But is there anyone here who can answer such questions , Paul ?

[Valentin_Pletzer]

Originally Posted by Paul Wilders

One should not jump to conclusions without knowing all the needed facts. And this does not in particular goes for NOD32, but for all Antiviruses tested .

regards,

paul

Hi Paul,

to make things easier. Here is the original e-mail from Andreas:

All products (in the "best" available Security Suite edition) were last updated on January 7, 2008 and tested on Windows XP SP2 (English).

First, we checked the signature-based on-demand detection of all products against more than 1 Mio. samples we've found spreading or which were distributed during the last six months (this means, we have not used any "historic" samples.) We included all malware categories in the test: Trojan Horses, backdoors, bots, worm and viruses. Instead of just presenting the results, we have ranked the product this time, from "very good" (++) if the scanner detected more than 98% of the samples to "poor" (--) when less than 85% of the malware was detected.

Secondly, we checked the number of false positives of the products have generated during a scan of 65,000 known clean files. Only products with no false positives received a "very good" (++) rating.

In case of the proactive detection category, we have not only focussed on signature- and heuristic-based proactive detection only (based on a retrospective test approach with a one week old scanner).
Instead of this, we also checked the quality of the included behaviour based guard (e.g. Deepguard in case of F-Secure and TruPrevent in case of Panda). We used 3,500 samples for the retrospective test as well as 20 active samples for the test of the "Dynamic Detection" (and blocking) of malware.

Furthermore, we checked how long AV companies usually need to react in case of new, widespread malware (read: outbreaks), based on 55 different samples from the entire year 2007. "Very good" (++) AV product developers should be able to react within less than two hours.

Another interesting test was the detection of active rootkit samples.
While it's trivial for a scanner to detect inactive rootkits using a signature, it can be really tricky to detect this nasty malware when they are active and hidden. We checked the scanner's detection against 12 active rootkits.

regards
Valentin

[C.S.J]

These massive tests are interesting at best.

Over 1 million new threats in the last 6 months I find extremely hard to believe

just how many of these are real threats that are circling around?

so I wouldn't worry Paul, about your beloved nod32. ( especially not on these huge tests anyway )

[MalwareDie]

Originally Posted by C.S.J

Only 2 fps from drweb?

Am I reading this right?

65 000 is quite a small number compared to av-comparatives' number of at least 10 million.

[Brian N]

I've never seen 10mil in a test at av-comp but whatever, Antivir is kicking ass.