AV-comparatives August2005 results released

[IBK]

Anti-Virus comparative of August 2005 is now released on www.av-comparatives.org

Please note that it is NOT allowed to provide the results/tables/documents on other sites or to put links to subpages. Please always link only to www.av-comparatives.org


I suggest to read the whole site and related documents/reports if you do not have already. Questions you may have are probably already answered in some documents.

I will visit various forums and maybe sometimes comment a bit. Today I will probably be mainly at the av-comparatives forum, the wilders forum and the rokop forum (german). Feel free to spread the word about the release of the August test results (I will post it probably only in the 3 mentioned forums).

For peoples from China that are not able to reach the orginal site, go to: http://www.itz.cc/ac

[Sputnik]

Thanks you IBK, amazing result by Symantec they really seem to work hard on their detection (signatures). BitDefender kinda dissapointed me, I thought they would have end up higher.

[Ned Slider]

Great!

Off I go to take a look...

Thanks :)

Ned

[likuidkewl]

Wow, I am surprised KAV got so many of the samples, but what really surprised me was Symantec's performance. Thanks for the food for thought.

[Ned Slider]

Yes, KAVs performance is remarkable.

NOD32 also looks like it's right up there with the very best now in terms of detection.

Ned

[Acadia]

Interesting stuff, thank you, IBK. 8)

Acadia

[xike]

Did this test include the buffer test, something like test of false positive or samples file with at least one antivirus scanner give false positive.......:)

[IBK]

A dedicated false positive test-set and results will be ready next year.

[Ned Slider]

Interesting again to see how many products are let down by their poor detection rates of trojans relative to other malware classes.

It would be interesting to see how dedicated trojan scanners (eg, a-squared, ewido etc) perform against the trojan subset. Any chance you could do such a test as a one off?

Ned

[richrf]

KAV is truly phenomenal. Thanks for posting the results.

Rich

[IBK]

{QUOTE-> Interesting again to see how many products are let down by their poor detection rates of trojans relative to other malware classes.
It would be interesting to see how dedicated trojan scanners (eg, a-squared, ewido etc) perform against the trojan subset. Any chance you could do such a test as a one off?
Ned <-QUOTE}

no, sorry, you will find the reasons in the faq i think. maybe i posted in past here on wilders (or av-comparatives or rokop; do not remember) some unofficial results of ATs, maybe u find them if you search.

[Diver]

Perhaps I need some more coffee, but I think there may be an error in the color coded graph in the report. The line for bitdefender does not seem to match the percentage results.

[IBK]

of course not. that graph shows the added samples since last test (test-set of february), not of the actual test-set. ;)

[Ned Slider]

{QUOTE-> no, sorry, you will find the reasons in the faq i think. maybe i posted in past here on wilders (or av-comparatives or rokop; do not remember) some unofficial results of ATs, maybe u find them if you search. <-QUOTE}

Thanks - didn't realise you'd already done some unofficial testing. I'll do a search and see if I can find it.

Thanks again,

Ned

EDIT: I assume this is the post you were refering to over at AV-Comparatives:

http://www.av-comparatives.org/forum/viewtopic.php?t=171

{QUOTE-> Just FYI: most AT programs reach more or less those results when tested against my malware test-sets:
OtherMalware: ~63%
Trojans: ~79%
Backdoors: ~88%
Total (backdoors, trojans, othermalware): ~83%

BTW: Ewido detects more than e.g. a².
<-QUOTE}

[QBgreen]

{QUOTE-> Wow, I am surprised KAV got so many of the samples, but what really surprised me was Symantec's performance. Thanks for the food for thought. <-QUOTE}

Nothing surprising with either of these products. Just check out their history!

[IBK]

@nedslider: yes, i think it was that one.

[Ned Slider]

{QUOTE-> @nedslider: yes, i think it was that one. <-QUOTE}

Thank you Sir :)

[Get]

Just an observation...let's say av-A scores 99% and av-B 98% than that still doesn't say what they missed. Could be av-A missed for the most part high geographic distribution threats and av-B for the most part low geographic distribution threats and therefor av-B could be much better irl when taken into account the geographical distribution and therefor the results, when the differences are small in percentage (don't know the exact margin of course), could be misleading.

[likuidkewl]

{QUOTE-> Just an observation...let's say av-A scores 99% and av-B 98% than that still doesn't say what they missed. Could be av-A missed for the most part high geographic distribution threats and av-B for the most part low geographic distribution threats and therefor av-B could be much better irl when taken into account the geographic distribution and therefor the results, when the differences are small in percentage (don't know the exact margin of course), could be misleading. <-QUOTE}

I often think about the geographic factor in AV tests, and it seems to be one of those unkowns.

{QUOTE-> Nothing surprising with either of these products. Just check out their history! <-QUOTE}

I have been following them, and 99% of the testbed is an extremely high number.

[iwod]

AntiVir did very well for a free AV. And if we uses av-comparatives as a guide NOD32 currently looks unbeatable.

Anyway, so far Antivir serves me well enough.

[Chubb]

Disappointed to see that Sophos only got 89.12% even with the new version 5.0.5~~

And, only F-PROT, KAV and McAfee got 100% in the macro virus test~~

[Firefighter]

DrWeb :'(

Best regards,
Firefighter!

[Don Pelotas]

{QUOTE-> DrWeb :'(

Best regards,
Firefighter! <-QUOTE}
Don't cry, Firefighter. It's still a good AV and 4.33 is on the horizon. :)

[Blackcat]

{QUOTE-> DrWeb :'(

Best regards,
Firefighter! <-QUOTE}
If you look at the Report file of these tests, particularly the present one, the rate that Dr Web is adding "missed" samples to their database is seemingly well behind the other vendors.

Maybe this small company needs to hire more analysts? :-\

Whereas Bitdefender and NOD, who are starting at the same origin as Dr Web, show much steeper rates in the detection of missed samples which coincided with their recent big updates before the present test.

Who would have thought 1-2 years ago that NOD and Norton would beat McAfee and Dr Web in trojan/backdoor detection!!

But don't worry, FF, IMO, Dr Web is still one of the best AV's ;)

[SDS909]

No test would convince me to replace Dr.Web on the majority of machines here. They require light and transparent AV's that have zero system drag, and it fits the bill nicely. Well, I might consider VBA32 since they added Dr.Web-Like settings for new files only.

[JRCATES]

Nice to see that McAfee still scored very high and did very well on these tests, since my McAfee subscription lasts another 9 months or so ;D

Being one of the top 4 or 5 AVs seperated in most instances by mere fractions of a percentage point in a highly competitive and ultra necessary field is definitely a good thing. Congrats to Kaspersky, NOD, BitDefender, Norton and McAfee for scoring the highest and best overall in these tests (and most importantly, to the users of these products)....

[waters]

Antivir seems to be pulling away in front, now of the free ones.The big updates did increase detection afterall.

[Firefighter]

{QUOTE-> Antivir seems to be pulling away in front, now of the free ones.The big updates did increase detection afterall. <-QUOTE}If I looked right, the best performers to increase detection rate in category "Total without DOS & other OS" were:

1. BitDefender
2. AntiVir
3. NOD
4. AVG

Best regards,
Firefighter!

[JRCATES]

Per IBK's request below, I have edited and removed material from this post:

Sorry IBK.....I merely took the link that you provided, and copied the results so that people visiting this particular thread in which you had pasted the actual link could referrence them in an easy fashion, that's all. I certainly wasn't trying to discourage anyone from visiting the particular link, because the percentages I removed didn't reveal anything in depth or detail, only the final test results.

[IBK]

In future, please avoid to provide the results on other sites/forums. :'(
EDIT: thank you JRCATES :)

[Atomic_Ed]

Wow looks like Avast! did pretty well overall and to me it reinforces my decision to purchase it. Appreciate the nice testing you do for us all.

[WSFuser]

wow these test results were amazing. all the AV (avg and sophos as an exception) got better than 90 percent. nod32 and norton are right up there with KAV. avast held its own as well.

[TAP]

Hi IBK,

Have you ever tested Fortinet FortiClient against Zoo malware? if so, could you please guide me about its overall detection rates (low, average, high)

The company I work using antivirusfirewall Fortinet FortiGate at network edge and we've evaluated so many corporate AV for our workstations such as AVG, FortiClient, NOD32, Panda, eTrust and we finally conclude that AVG, FortiClient, eTrust are the best 3 solutions for us but we don't know about FortiClient's overall detection rates.

Thanks

[I who know nothing]

I sometimes wonder which scores people actually read when they look at the results and make their conclusions. Relatively high overall scores are one thing but if you actually look at the scores for individual categories you may get a different impression altogether eg relatively low AntiVir score on scripts 62% and windows virus 76% , AVAST on scripts 62% and trojans 73%, AVG on scripts 33% and trojans 58%. Reason I use KAV is because it consistently scores 99 plus in every category.

[I who know nothing]

{QUOTE-> Wow looks like Avast! did pretty well overall and to me it reinforces my decision to purchase it. Appreciate the nice testing you do for us all. <-QUOTE}
You dont mind it scored 62% script and 73% trojan detection?

[ErikAlbert]

I who know nothing,
If you have KAV + McAfee on your computer, then you have the maximum protection.
KAV for all kinds of malwares, except the first and last one.
McAfee takes care about the rest.

[WSFuser]

i dont recommend two AV, better just have KAV and ewido for AT and backup.

[ErikAlbert]

{QUOTE-> i dont recommend two AV, better just have KAV and ewido for AT and backup. <-QUOTE}
I agree, but this thread is about AV's only.

[TAP]

{QUOTE-> You dont mind it scored 62% script and 73% trojan detection? <-QUOTE}

If I'm not wrong, AV-comparatives on-demand tests AVs against so-called Zoo malware that some of them (or most) are rarely circulating in the wild and you are rarely (or may not) encounter them in real life, if you're not trying to infect yourself by downloading/clicking everything you see on the internet so I think even AVG Free or eTrust EZ would offer an adequate protection.

I believe that AntiVir, AVG, Avast, Trend Micro, Sophos, eTrust will offer the best protection against today's most dangerous malware even it can't score 99% plus in every category.

[WSFuser]

{QUOTE-> I agree, but this thread is about AV's only. <-QUOTE}
then what do u mean by "kav + mcafee"? just have both and use one resident. theyre both high end as it is.
{QUOTE-> If I'm not wrong, AV-comparatives on-demand tests AVs against so-called Zoo malware that some of them (or most) are rarely circulating in the wild and you are rarely (or may not) encounter them in real life, if you're not trying to infect yourself by downloading/clicking everything you see on the internet so I think even AVG Free or eTrust EZ would offer an adequate protection.

I believe that AntiVir, AVG, Avast, Trend Micro, Sophos, eTrust will offer the best protection against today's most dangerous malware even it can't score 99% plus in every category. <-QUOTE}
i agree, but some people (high risk surfers) do need (or want) teh extra protection provided by an outstanding AV like kav.

also looking at the individual scores rather than teh overall one is different, i may just have another look at the results.

[Stefan Kurtzhals]

{QUOTE-> but if you actually look at the scores for individual categories you may get a different impression altogether eg relatively low AntiVir score on scripts 62% and windows virus 76% , AVAST on scripts 62% and trojans 73%, AVG on scripts 33% and trojans 58% <-QUOTE}

If people had an idea how AV programs would work technically they would understand those particular scores... :)

The test shows that there is only one choice and that is KAV. However, scanning some "exotic" malware collections (from different geographical regions), even KAV doesn't get above 60% detection sometimes (while other AV's don't even reach 20% then). KAV is the fastest in working up the detection rate as soon they have the collection though.
Alot of malware we are getting in from our automated analysis system is completely undetected, by all AV programs. On those, NOD32 has quite a good chance to catch because of the excellent heuristics. VBA32 also sometimes catches some sample because of the good variant detection.

In the end, there is no real 100% protection. Even with a fully patched XP SP2 system, with Process Guard (free), Prevx (free) and Firefox, latest JRE I was able to get a goat machine infected visiting some "nasty" web pages. Of this adware/malware, KAV detected the most by far - but it missed a few still.

[Graystoke]

Geez. Here I am happily rolling along running McAfee VS 10, and now I see NAV out performs it.

I am happy with what I've got, I don't want to switch. I am happy with what I've got, I don't want to switch. I am happy with......................... :P

[waters]

Will there be the results for the other ones you have tested like panda,avk,etc

[IBK]

No, sorry, they are only for the av companies.

[bathisland]

This just makes me happy that I am one of KAV's very satisfied customer ;D

[waters]

Why did you show results for a seperated test for 03,2005,and not for sept.
You tested another 5,and showed results.In it panda and fsecure got advanced+

[IBK]

That was an exceptional gift.
Next year some of them will be included in the regular detailed tests.

[ErikAlbert]

{QUOTE-> then what do u mean by "kav + mcafee"? just have both and use one resident. theyre both high end as it is. <-QUOTE}
If you look at the detection rates of KAV and McAfee and you compare them with the rest, you will discover that KAV is the very best and what is lesser good in KAV will be compensated by McAfee, which is also the very best in some malware types (the first and the last one).
So KAV and McAfee TOGETHER beat the rest of the AV scanners in detection rates of all malware types.
I had it all in full detail to prove it, but my post was removed, because I didn't know about the rule, that nothing of AV-comparatives results can be used in other forums, including Wilders.

So if I was willing to pay for security software, I would buy KAV as main scanner and McAfee as scanner on demand.
One thing is sure you can't buy them all, which also means that your computer isn't necessarily clean after running KAV and McAfee, because most scanners have a different fingerprint database in quantity and identity.
If you think, I'm wrong just say it. I'm open for any other vision.
After all security isn't my strength :)

EDIT:
Of course this is only valid until the next test in November 2005.
Maybe AVG will be the big star in November, who knows 8)

[Ned Slider]

^^ Agreed, both KAV and McAfee are excellent products, but you do need to be careful when making statements like the above. I ran a very quick test over my very small (relative to AV Comparatives) virus collection and even after running KAV followed by McAfee, things were still missed - even from my relatively small sample collection (all collected from infected machines, so these are real samples that are in circulation).

As is everyone else, I'm hunting for the holy grail that just isn't out there - one or two products that when used in combination will catch everything. It just doesn't exist. Even running KAV, McAfee, BitDefender, Ewido and a-squared sequentially over my collection reveals samples remaining that are currently detected by none of them and these are all class leading products.

Just my 2 cents :)

Ned

[ErikAlbert]

{QUOTE-> ^^ Agreed, both KAV and McAfee are excellent products, but you do need to be careful when making statements like the above. I ran a very quick test over my very small (relative to AV Comparatives) virus collection and even after running KAV followed by McAfee, things were still missed - even from my relatively small sample collection (all collected from infected machines, so these are real samples that are in circulation).

As is everyone else, I'm hunting for the holy grail that just isn't out there - one or two products that when used in combination will catch everything. It just doesn't exist. Even running KAV, McAfee, BitDefender, Ewido and a-squared sequentially over my collection reveals samples remaining that are currently detected by none of them and these are all class leading products.

Just my 2 cents :)

Ned <-QUOTE}
I think you didn't read my post completely. I mentioned it that KAV + McAfee doesn't guarantee a clean computer.

[Ned Slider]

{QUOTE-> I think you didn't read my post completely. I mentioned it that KAV + McAfee doesn't guarantee a clean computer. <-QUOTE}

Noted :)

[SDS909]

{QUOTE-> Will there be the results for the other ones you have tested like panda,avk,etc <-QUOTE}

AVK uses Kaspersky bases and Bit Defender. As such, it will score superior to the base KAV products with both engines activated and on full settings.

Not to mention AVK is far far more bug free than KAV.. I refuse to use KAV ever again, because the 4 times i've tried it, it always caused system instabilities, was buggy, or otherwise I ended up formatting that PC to clean it off. Not to mention the fact that KAV has some of the most useless fluff in their bases and i've found the KAV people to be rude at times.

It's a product I avoid, regardless. AVK however I will/do use on client machines because it is a nice product.

[WSFuser]

{QUOTE-> Geez. Here I am happily rolling along running McAfee VS 10, and now I see NAV out performs it.

I am happy with what I've got, I don't want to switch. I am happy with what I've got, I don't want to switch. I am happy with......................... :P <-QUOTE}
then just stick with mcafee. its a very good AV and NAV only outpeformed it by very little. also u shouldnt use such tests for choosing an AV.

[TeknO]

Summary Results;

TeknO

Read the first post in this thread please. AV-Comparatives does not want their test results published on other sites.

[IBK]

just a short reminder (see first post or on the website):
Please note that it is NOT allowed to provide the results/tables/documents on other sites or to put links to subpages. Please always link only to www.av-comparatives.org

[TeknO]

{QUOTE-> Summary Results;

TeknO

Read the first post in this thread please. AV-Comparatives does not want their test results published on other sites. <-QUOTE}
OK I changed it.

Summary Results;
Kaspersky is the best in the test...

;D ;D

[likuidkewl]

IbK- can you clarify the point you are trying to convey in section 6 "Non detected samples in the test bed"

I am confused in the wording of the note under the graph,
I see that 68% were detected by all 12, ok easy enough. But the graph break down is confusing. (NOT a single scanner!) means that they were flagged by one of the 12 but not the same one? Soory but I just can seem to break it down.

//EDIT//

This is what I have so far:

286,396.28 <-- 68% of samples, in numerical value
421,171 <-- total number of samples
----------
134,775 <-- total number of samples not caught by all 12 scanners

Maybe you could give a 1-12 just numbered list to simplify it?

//EDIT #2//
So no sample made it by all 12 without getting flagged, right?

[JRCATES]

{QUOTE-> then just stick with mcafee. its a very good AV and NAV only outpeformed it by very little. also u shouldnt use such tests for choosing an AV. <-QUOTE}

Exactly. The difference is so insignificant that the cost, frustration and aggravation of "switching" a product simply because it was outperformed by fractions of percentage points wouldn't make any sense.

Besides, who's to say that results like these don't "inspire" a company to improve their product even more.....while others might slack off a little.....and 3 to 6 months from now we have completely different results. After all, we had different results from May to August ;)

[IBK]

I think you got it likuidkewl. :)

@JRCATES: you mean February to August. because may and november are retrospective tests...

OT:
BTW: I know I am annoying anyone when I write things that you ALL already read (like suggested everywhere on the site, bold, underlined, in color, etc.) in the test report, but maybe some of you forget it; here a short quote:
"Products belonging to a category can be considered to be as good as the other products in the same category regarding the on-demand detection rate. All products in the ADVANCED+ category offer very high level of on-demand detection. Selection of a product from this category should not be based on detection score alone. The quality of support, easy of use and system resource use should be considered when selecting a product."

[iwod]

{QUOTE-> I sometimes wonder which scores people actually read when they look at the results and make their conclusions. Relatively high overall scores are one thing but if you actually look at the scores for individual categories you may get a different impression altogether eg relatively low AntiVir score on scripts 62% and windows virus 76% , AVAST on scripts 62% and trojans 73%, AVG on scripts 33% and trojans 58%. Reason I use KAV is because it consistently scores 99 plus in every category. <-QUOTE}

i look at all of them. Those are data only. And how one analyse or interpret is another thing. I know AntiVir does relatively bad in Win Virus and Scripts. I hope they improve as well.

[The Hammer]

{QUOTE-> then just stick with mcafee. its a very good AV and NAV only outpeformed it by very little. also u shouldnt use such tests for choosing an AV. <-QUOTE}I must respectfully disagree. When choosing an Av a person who does not work in the field for a living either building ,repairing, installing networks or computers ( in other words the average Joe) must gleam information from whatever sources he can. Many people with out a techy background want the best protection possible even if it's only a few percentage points. I like NOD's zero day protection but originally got NOD because I have a older machine and wanted to limit drag on my system. Using this test to choose an Av is valid in my opinion because after all many people consult statistics when choosing other products like a cars front impact crash test rating for instance.

[WSFuser]

{QUOTE-> I must respectfully disagree. When choosing an Av a person who does not work in the field for a living either building ,repairing, installing networks or computers ( in other words the average Joe) must gleam information from whatever sources he can. Many people with out a techy background want the best protection possible even if it's only a few percentage points. I like NOD's zero day protection but originally got NOD because I have a older machine and wanted to limit drag on my system. Using this test to choose an Av is valid in my opinion because after all many people consult statistics when choosing other products like a cars front impact crash test rating for instance. <-QUOTE}
i doubt the "average joe" would know about these kinda sources and tests in the first place. however just like u can test drive a car, people should also make an attempt at trialing antiviruses and other software. tho i guess that would cause problems. and lastly i feel u discover more through personal use of a program than by just researching it, but both are necessary.

[The Hammer]

{QUOTE-> i doubt the "average joe" would know about these kinda sources and tests in the first place. however just like u can test drive a car, people should also make an attempt at trialing antiviruses and other software. tho i guess that would cause problems. and lastly i feel u discover more through personal use of a program than by just researching it, but both are necessary. <-QUOTE} I found out about this site through a mainstream site PCmag and was not referred to it by any "geek guys" and I mean to use that term in a good way. I'm sure I am not the only one to have found my way here without help from anyone "in the field". Believe me I'm as average a Joe as your going to find. ;D

[WSFuser]

{QUOTE-> Believe me I'm as average a Joe as your going to find. ;D <-QUOTE}
do average Joes always visit Wilders and read PC Magazine? :P

[The Hammer]

;) {QUOTE-> do average Joes always visit Wilders and read PC Magazine? :P <-QUOTE}They may start with PC mag at the begining and come across a link to some other sites such as Wilders as they attempt to learn more about computers and security. Especially if they watch the news like CNN where they recently devoted a lot of valuable air time to a trojan outbreak which downed a number of their computers at their headquarters in Atlanta.

[JerryM]

As I am not very sharp in the area of computers, I do rely heavily on such tests. However, I am not wedded to them, and have been using Bit Defender, not V9.0, for several months.
It is a couple of percentage points below the top, but I accept that. I have been pleased with BD, and do not intend to change. However, if I had no good AV I would choose KAV, or possible NOD.

Using BD and Ewido along with LnS I have a high degree of confidence in my set up. If I were using a AV that scored in the low 90s I would change when my license ran out.

I recognize that some AVs do not run well with some combinations on a computer, and that would be the deciding factor. I have an impression that KAV is touchy in that way, but that is only an impression gained from reading problems here. I may be wrong.

Jerry

[Graystoke]

{QUOTE-> Quote:
Originally Posted by Graystoke
Geez. Here I am happily rolling along running McAfee VS 10, and now I see NAV out performs it.

I am happy with what I've got, I don't want to switch. I am happy with what I've got, I don't want to switch. I am happy with......................... :P <-QUOTE}






{QUOTE-> then just stick with mcafee. its a very good AV and NAV only outpeformed it by very little. also u shouldnt use such tests for choosing an AV. <-QUOTE}


No plans to switch. It was just my attempt at a little humor ;) ;D

[profhsg]

Andreas:

I've read your very informative document on testing procedures and FAQs. I know that your margin of error is < .1% and that one should not choose an AV only on the basis of detections percentages. I still have a question which you might choose to answer. Is there a level of difference in detection rates which you consider insignificant? In other words if product X has a 98% and product Y has a 98.3% rate (the difference is obviously outside the margin of error) is there any real world significance between the detection rates of the two products?

Thanks for your great work.

[IBK]

No, that is why the levels were introduced and why I say "Products belonging to a category can be considered to be as good as the other products in the same category regarding the on-demand detection rate." The exact numbers are firstly of interest for the AV companies and other experts, there were some ideas if in future the users should only see which levels were reached, like some other testing sites does, but personally I do not want to do that; I prefer to keep showing them and to try to explain the users how to interpret the results... even if I sometimes read in forums things that let me suppose that only few peoples read the whole report or the comments under the results.