http://www.av-comparatives.org/

[FYI]

http://www.av-comparatives.org/

[meneer]

;D
-{ Quote: "In order to get tested by us, products must fulfill some criteria. The 3 main criteria are:

- The product must use only one (own) scan-engine
- The product must have a 100% ItW-virus detection
- The product must detect a given minimum of Zoo-samples " }-

So, in order to get tested, it must have been tested ???
Perhaps this site is okay, but this makes it look stupid :-\

[Trans]

If the tests there have any value,
interesting results for Panda and NOD...

[FanJ]

Any info on the test set?
Any info with respect to those statistics about whether in-the-wild and/or zoo-viruses?
Any info about how the test was performed?

[IBK]

Hello,

yes, there were preliminary tests in order to discover which products can be get tested, as FOR EXAMPLE not every company would like to submit their products if they know that they will get poor results. Other reason is that it would require also to mcuh time to test products that does not fulfill the conditions (those are just 3 of about 10 conditions).
I was also a bit impressed from the results of Panda, I expected lower results, but it seems that Panda increased in last months their detection rate quite much. (I tested Panda Platinum Internet Security; Panda Titanum and Platinum AV has a bit lower detection rates as those does not detect for example Hacker/virus tools, spyware etc.).
I expected higher results for NOD32, as I used also the /AH option. NOD32 apparently removed from their databases some old dos viruses, I think this is also one reason why they got such a low total result; I am sure ESET will do their best in order to get better results next time. Anyway all tested products are already very good products; in my opnion, NOD32 has still the best heuristic of all, but I can say this for sure only after the next test.
Regards,
IBK

P.S.: I do not visit often this forum, so I will not be able to respond to all questions here.

[MikeBCda]

Interesting. I completed (or tried to) their survey while I was there, and someone should get the word to them that it's very poorly designed.

If you tick "no" to the question about whether you ever got a virus even with a-v running, and naturally skip the following question about how you got the virus, it kicks back to you that there's an answer missing.

And the wording about very old DOS viruses ("Do you pretend you're protected ...?") was a little insulting.

[nameless]

Does anyone ever, like me, wonder if the people who carry out these tests notify the vendors about the malware their products missed? Or, if they don't... why they don't?

Or, how about why every test rates KAV very, very well, yet KAV doesn't have a stellar VB100 track record? I know that the problem with KAV where VB100 is concerned has to do with false positives, but this tells me that some aspects of the VB100 test are pretty damned stupid.

[Kay Maier]

Good day!

The difference with these tests is the following one: Virus Bulletin publishes their testset (names) before they perform tests. Therefore, Eset can add all of these viruses easily. The testset VB uses is also rather small.

Other testers, such as Clementi or Marx do not publish the testset in advance and it includes much more samples. Virus Bulletin has only 400 different viruses in their testset in 10.000 files or so. Clementi and Marx seems to use much more viruses for test (50.000? in 250.000 infected files?). At least from Marx I know that he sends out a collection of all missed viruses after the test is finish and the results are final.

Regards,
Kay

[illukka]

-{ Quote: " quoting: nameless link=board=24;threadid=22807;start=0#msg135691 date=1077736248]
this tells me that some aspects of the VB100 test are pretty damned stupid.
" }-

yeah vb only tests if a scanner has a signature for those viruses that currently are in the wild and if that signature is strong enough not to produce false positives. it has nothing to do with overall detection abilities.

[IBK]

I follow more or less the same principles as Marx has and the same fair conditions as VB. ;)

About the survey: Yes I know, it has some problems; but believe me, I got such a headache to make it, that I decided to make it online as it is, as otherwise I would had to delay for another week the publishing of the results. So, please accept the survey as it is, I am not going to spend more time to fix it (there are more errors in the survey questions than you named). Anyway if you read on the top, you must fill out all questions, so i do not understand why you wonder ;)

P.S.: NOD32 is a very good scanner, even if you maybe think that the results are low. NOD32 protects you against all ITW-viruses and is a fast scanner. As I said, the results showed are just one little aspect of a scanner, there are still many other interesting aspects that makes NOD32 very attractive. :)

[Bender]

Kay Maier ----- ROFL

[sir_carew]

IBK,
ESET isn't delete any viruses/trojan/exploit (malware) from their database even if it's a very old DOS virus.

[Firefighter]

To Bender from Firefighter!

Kay Maier isn't so wrong what he/she wrote. Here are those polymorphic virus names in VB 6-2002 and 6-2003, can u see there some big differencies?

Polymorphic Viruses VB 6-2002 winXP (42 different virus names):

ACG.A 174, ACG.B 90, Alive.4000 500, Anarchy.6503 500, W97M/AntiSocial.E 50, W97M/AntiSocial.F 20, Arianna.3076 500, Baran.4968 500, XM/Compat.A 50, Code.3952:VICE.05 500, Cordobes.3334 500,
Cryptor.2582 500, W32/CTX 84, Digi.3547 500, DSCE.Demo 500, W32/Fosforo 61, Girafe:TPE 500,
Gripe.1985 500, W97M/Groov.B 50, Mad.3544 500, W32.Magistr. 17, Win95/Marburg.8590 744,
Groove and Coffeeshop 500, MTZ.4510 500, Natas.4744 500, Neuroquila.A 500, Nightfall.4518.B 500, One_Half.3544 500, Pathogen:SMEG.0_1 500, PeaceKeeper.B 500, Russel.3072.A 500, SatanBug.5000.A 500,
Sepultura:MtE-Small 500, Win95/Sk.7972 7, Win95/Sk.8044 21, SMEG_v0.3 500, Spanska.4250 500, W97M/Splash.A 100, W97M/Service.A 100, Tequila.2468.A 500, Uruguay.4 500, W32/Zmist.D 43


Polymorphic Viruses VB 6-2003 winXP (43 different virus names):

ACG.A 174, ACG.B 90, Alive.4000 500, Anarchy.6503 500, W97M/AntiSocial.E 50, W97M/AntiSocial.F 20, Arianna.3076 500, Baran.4968 500, XM/Compat.A 50, Code.3952:VICE.05 500, Cordobes.3334 500,
Cryptor.2582 500, W32/CTX 84, Digi.3547 500, DSCE.Demo 500, W32/Etap 29, W32/Fosforo 61, Girafe:TPE 500, Gripe.1985 500, W97M/Groov.B 50, Mad.3544 500, W32.Magistr. 17, Win95/Marburg.8590 744,
Groove and Coffeeshop 500, MTZ.4510 500, Natas.4744 500, Neuroquila.A 500, Nightfall.4518.B 500, One_Half.3544 500, Pathogen:SMEG.0_1 500, PeaceKeeper.B 500, Russel.3072.A 500, SatanBug.5000.A 500,
Sepultura:MtE-Small 500, Win95/Sk.7972 7, Win95/Sk.8044 21, SMEG_v0.3 500, Spanska.4250 500, W97M/Splash.A 100, W97M/Service.A 100, Tequila.2468.A 500, Uruguay.4 500, W32/Zmist.D 43


Yes, there was only W32/Etap 29 new in VB 6-2003 polymorphics that wasn't in 6-2002 and the VB 8-2003 NetWare has exactly the same virusnames that were in VB 6-2003 WinXP polymorphics!

You can check my lists from official VB published PDF issues 6-2002 and 6-2003 if u want.

Besides that "in the Wild" list is more or less known before VB is testing those av:s!


PS. There were actually some 1 600 different virus names in VB 6-2003 WinXP test that measured macro, polymorphic and standard viruses and those files were in about 21 000 samples, so that Kay Maier's 400 and
10 000 may be a bit inaccurate, but most of them all were more or less known before the test.


"The truth is out there, but it hurts!"

Best regards,
Firefighter!

[IBK]

Thanks for letting me know. When I will have time I will check whats up and report directly to ESET if I find a reason why about 6 months ago NOD32 appaered to detect more DOS viruses than yet.

[mrtwolman]

with all the respect to the author of the test, based on the information on the test published on the site i'd label it as not too proffesional.

My reasons:
1. only AV with one scan engine can take part in test. Why?
2. there are no data on test sets. No data on origin of the viruses, no data on their ability to replicate... Kinda in the old good virus.gr test style.
3. what is best possible setting for an antivirus? As long as the exact settings are not published, it gives no relevent information
4. there is no way to confirm the test results

I see some problems with methodology here....

[IBK]

Hello,

1. because I tested primary the scan engines not the products. F-Secure or AVK would of course get better results than the others, as they use for example the F-prot engine and the KAV engine.
2. Data of test sets is known to the AV producers. Normal users does not need such information and of course we are able to replicate samples. AV companies agreed to be tested be us, we are not comparable with amatorial underground tests like virus.gr ; anyway anyone can prefer the tests that he better like.
3. the best possible settings are the best possible settings; for example all files, deep heuristic etc. (best settings are also known to the AV producers and confirmed by them if unclear).
4. AV companies are in contact since long time and test results can be confirmed as they can check theirself the results.
Methodology follows the standard testing procedures, like VB or Marx does. Anyway I will talk to some people about the methodology most probably at the next EICAR conference in Luxembourg. See you there!

Regards,
Andreas

[Firefighter]

To everyone from Firefighter!

What do u think of the value of certain test when it has only 1 (uno) new virusname in one of their detecting categories 14 months later? In my vocabulary those tests are some kind of facelifts, that has been done to appear more acceptable among the ignorants!

In other words, VB is testing 14 months later only that, does those tested av:s detect 1 new virus or not in their polymorphic sample database.

Where is the ROFL now?


Best regards,
Firefighter!

[mrtwolman]

-{ Quote: " quoting: IBK link=board=24;threadid=22807;start=15#msg135979 date=1077794671]
1. because I tested primary the scan engines not the products.
" }-
With regard to above statement - maybe the "scanning engine test" would be more appropriate than antivirus test....

-{ Quote: " quoting: IBK link=board=24;threadid=22807;start=15#msg135979 date=1077794671]
2. Data of test sets is known to the AV producers. Normal users does not need such information and of course we are able to replicate samples. AV companies agreed to be tested be us, we are not comparable with amatorial underground tests like virus.gr ; anyway anyone can prefer the tests that he better like.
" }-
Anyway, the statement "all samples are replicable and their replicability was verified it in this and that way" would improve the impression of the naked numbers presented in the tables. Another statement in the style of "list of tested files available upon request" would boost image of the test too.

-{ Quote: " quoting: IBK link=board=24;threadid=22807;start=15#msg135979 date=1077794671]
3. the best possible settings are the best possible settings; for example all files, deep heuristic etc. (best settings are also known to the AV producers and confirmed by them if unclear).
" }-
Nevertheless, couple of bytes of extra typed characters explaining the setting would not cause any harm.

-{ Quote: " quoting: IBK link=board=24;threadid=22807;start=15#msg135979 date=1077794671]
Methodology follows the standard testing procedures, like VB or Marx does. " }-
I just wanted to point to the matter of fact that known methodology of test will greatly boost the credibility. One of reason why are VB test so respected is that whoever who has enough know how is able to verify test results on his own.

Maybe in the future you could add hard poly detection test - tent of thousands of poly vurises in huntreds of generations. The results sometimes could be very very interesting.... 8)

[Firefighter]

To mrtwolman from Firefighter!

U wrote,

"One of reason why are VB test so respected is that whoever who has enough know how is able to verify test results on his own."

In my mind there is a bit difference between respected and reliable. VB may be reliable but to test one new virus in polymorphic category 14 months later does not make VB as respected in my vocabulary!


"The truth is out there, but it hurts!"

Best regards,
Firefighter!

[mrtwolman]

-{ Quote: " quoting: Firefighter link=board=24;threadid=22807;start=15#msg136015 date=1077803393]
In my mind there is a bit difference between respected and reliable. VB may be reliable but to test one new virus in polymorphic category 14 months later does not make VB as respected in my vocabulary!
" }-
You have right to have your opinion as well as I have right to have mine. With all the respect, there is no test on this planet better than that of VB. Mine explanation of respected is a bit different - there was no "issue" with VB test since the very beginning of the testing. Sure, tests of VB are limited in some aspects but are bringing valuable informations. Are replicable. Have clear methodology. Use real replicating samples. Are performed in the same way all the times. Just try to mention any test able to hold the same level with VB 8)

[Firefighter]

To mrtwolman from Firefighter!

U wrote,

"Sure, tests of VB are limited in some aspects but are bringing VALUABLE informations."

So, what kind of value does that kind of test have if an av detects one (1) polymorphic virus 14 months later or not?

Detected YES or NO , but what can u assume about the av:s reliability - nothing!

I'm still ranking that kind of VB tests at least as missleading!


"The truth is out there, but it hurts!"

Best regards,
Firefighter!

[Bender]

Firefighter ----- a ROFL for you too! I will not waste my time trying to explain where you are wrong, because your historic posts tell me you do not want to know, only to argue with a closed mind.

[Technodrome]

Lets stay on topic folks. ;)



tECHNODROME

[mrtwolman]

-{ Quote: " quoting: Firefighter link=board=24;threadid=22807;start=15#msg136015 date=1077803393]
In my mind there is a bit difference between respected and reliable. VB may be reliable but to test one new virus in polymorphic category 14 months later does not make VB as respected in my vocabulary!
" }-
Before I will rest my case in this discussion, I kindly ask you to answer to my questions:

1. Can you name some other antivirus test that matches that of VB ?
2. Can you name one new polymorphic virus which was ITW in last 14 months ?

And now I rest my case.

[Firefighter]

To Bender from Firefighter!

My history in here does not have anything to do with this topic. I am happy to see some errors in my post just now but u can't just find them!

Besides, even I can make mistakes occasionally, but I admit those when some has real facts to show to me. I'm only now showing these things like the others were showing them to me lately when I said some other av-tests but VB as thrustworthly!


"The truth is out there, but it hurts!"

Best regards,
Firefighter!

[IBK]

IMPORTANT!!! I took a deeper look on why NOD32 get those results. I found the reason; the person who tested NOD32 made a fatal error (damn >:(). The fixed results will be up in some hours. Please update then your data. It was really lucky that I discovered this before the 1st March, as the official results have to be up at that time.

[Firefighter]

To IBK from Firefighter!

Don't worry about your mistake! Even VB has made some mistakes recently, McAfee 6-2002 and Avast 2-2004 were awarded afterwards to get their 100% VB Awards.


"The truth is out there, but it hurts!"

Best regards,
Firefighter!

[IBK]

I am really much worry about this fact, even if this is the first public test. Well, I will learn from the mistakes. The fixed results should now be up.
As I just noticied this, I will have to think if I will test next time in August again NOD32, as if it has problems to detect virality in non-executable extension, it does not fulfill the conditions. But as NOD32 is a very popular scanner and everyone want to see its resutls, I will see if I can/will change some conditions in next months.

[Firefighter]

To IBK from Firefighter!

The zipped results were not corrected but the online results were new!

Best regards,
Firefighter!

[IBK]

Many thanks for letting me know! Now it should be all up (really) ;-)

[NOP]

Hi all,

Just my curiousity, if you read an archives ( PDF ) that published on VB website and look at comparative review you'll see something as following.

ItW Overall 100.00% Macro 100.00%
ItW Overall (o/a) 100.00% Standard 99.73%
ItW File 100.00% Polymorphic 99.82%

I want to know that :

Macro, Standard and Polymorphic virus sample are the ITW viruses or Zoo or anything else?

what is the meaning of " (o/a) " ?

Sorry for stupid question, Thanks. :)

[Firefighter]

To NOP from Firefighter!

Macro, Standard and Polymorphic samples are Zoo viruses in VB tests, because how it is possible that there were almost the same viruses 14 months in polymorphic samples?

The detecting rates are weighted, not real detecting rates.

"The truth is out there, but it hurts!"

Best regards,
Firefighter!

[nameless]

Oh.... Somone made a boo-boo, so now NOD32 rises from medicocrity all of the sudden? This inspires confidence in the test, to be sure. What, did Eset's lawyers contact you?

[Firefighter]

To nameless from Firefighter!

U have to be kidding!

Eset is still a minor player in av-business, what do u expect Sophos and Trend Micro representatives to do because they were poorer than NOD in total without DOS and other OS or backdoors and trojans in AV-comparatives.org 2-2004 tests?


"The truth is out there, but it hurts!

Best regards,
Firefighter!

[IBK]

Again: maybe you did not understood exactly one thing: ALL AV products that are listed/tested on the website are ALL VERY GOOD SCANNERS! All have very high detection rates, but someone must everytime be "the last" if you make a ranking. But on the website is no ranking to see, as all products are good and detection rates changes fast - ESET has the samples and they constantly add samples. If I would test the programs in one month, the "rankings" as you are making would be probably totally different. There are no "loosers" in our tests. All are winners.

And no, ESET's lawyers did not contact me. Anton Zajac was sick last weeks, so probably this is the reason why he did not noticied me about this issue before I made the results public. I had to discover it without Zajac :-(
Good that it is fixed now.

[JimIT]

-{ Quote: " quoting: Firefighter link=board=24;threadid=22807;start=30#msg136159 date=1077827050]
To NOP from Firefighter!
The detecting rates are weighted, not real detecting rates.
Best regards,
Firefighter!
" }-

This statement needs clarification, FF. Can you expound?

Thanks! ;)

[Firefighter]

To JimIT from Firefighter!

About VB detecting rates look at this link.

http://www.virusbtn.com/old/comparatives/Win95/199801/protocol.html

For this reason, one av may have let's say 225 misses in polymorphics, but it actually missed one single virus. In the same time an other av may have let's say 25 misses, but it actually missed 3 viruses and it's detecting rate is lower.


"The truth is ou there, but it hurts!"

Best regards,
Firefighter!

[Paul Wilders]

Andreas,

Willkommen ;). Erstmal ne glückwunsch für die geleistete arbeit. So ein projekt ist vielumfassend ohne frage.

Wenn da neues zu berichten gibt - bitte schreibe es hier im forum.

grusse,

Paul

Andreas, welcome. First my congrats for an exhaustive job done - these kind of tests are a big job without any question. As soon as you have anything new to report, don't hesitate to post over here

[IBK]

I write in english so everyone can understand me.

@Wilders and all others: yes, there is something new to notify you

-> I removed some more bad samples and fixed the results again. As I predicted, the results did not changed that much even if I removed the bad samples. Some samples were in the meantime added from some av companies - anyway they are bad and needed to be removed; they were probably added because many users submitted them and asked for detection, even if they do not work properly.

@ESET and all other AV companies: please update the PDF and the other data that is only accessible for AV related companies.

If more bad samples are found, I will notify it under the results, but not change again the results.

[Paul Wilders]

@Clementi: I'll inform Anton Zajac ;)

regards.

paul

[Firefighter]

To IBK of Firefighter!

Can we expect that all those av:s that were tested are able to detect all those tested infected files after a while?


"The truth is out there, but it hurts!"

Best regards,
Firefighter!

[IBK]

yes you can :-) but it depends also from the AV company; if they give to the samples low priority, it can take loong time while before they detect all; some AV does in the meantime already detect over 50% of their missed samples. Even if the scan was done just 3 weeks ago, the data is atm again outdated, as detection rates changes fast.

[Firefighter]

To IBK from Firefighter!

Thanks for Your answer. I have still an other question. I am not sure how many different infections there were in Your test including all categories, can You advice me a bit, Please?


"The truth is ou there, but it hurts!"

Best regards,
Firefighter!

[IBK]

Thats quite hard to answer, as to determine what is called a different infection is not always easy. I could refer to various names given by AVs (e.g. over 44000 different names by KAV in total), but thats not exactly at all. - and i prefer to do not write all details here to the public...

[steve1955]

In most "sciences"the person discovering something"new" names that finding(in this case viruses etc)and all other researchers adopt that given name. Isn't it about time the AV vendors/industry fell into line with this practice,it would stop the same piece of malware having multiple names!

[mrtwolman]

-{ Quote: " quoting: steve1955 link=board=24;threadid=22807;start=30#msg137172 date=1078002690]
In most "sciences"the person discovering something"new" names that finding(in this case viruses etc)and all other researchers adopt that given name. Isn't it about time the AV vendors/industry fell into line with this practice,it would stop the same piece of malware having multiple names!
" }-
That is near impossible. When worm or virus appears, in AV companies are getting sapples in almost no time. Who is the first, who has the right to give name.... I am afraid the things will work as now for a very long time.... Anyway, maybe you noticed after some time they almost in every case meet some common name or two or three for a worm or virus :P

[steve1955]

The problem about "who was first?" applies to all discoveries but eventually one "name " is adopted this doesn't seem to happen with the Av companies there are still viruses from the past that have more than one name,and they shouldn't(good job the medical profession dont operate the same way:-your holiday would be over whilst trying to find out which jabs you needed/were up to date to protect you against diseases with multiple names!)