New AV-Test.org malware testing (Avira finished 1st, CA eTrust finished last)

The thing is that I'm pretty sure not everyone knew WHEN Andreas was going to publish the test results. Therefore, 15th and 16th May was in the conference, 17th May all analysts get home, and 18th May products were updated. How much time to add samples? One day, or maximum 2. The fact is that major analysts were at the conference, it is very difficult to send 15GB worth of samples to other company analysts via email, so the samples were likely received via disks or through download links (unlikely because the collection was already being distributed), so most likely it was distributed on DVDs.

So in the end, analysts had only the day of 17th and maybe some small part of 18th may to add definitions. How many would they add in that period? It would be very difficult for such additions to cause any significant impact on the test results.

 

Oh?

Sorry then, my mistake
Thanks for clarifying.

 

The others also did get it, you know, with no effort on their part. Its not like VBA32 had to go request those samples from Marx, from what I see Marx gave those samples of his own will.

 

ren,

you bring up a very interesting point. Why would Andreas Marx share samples at the conference before the actual test itself?

1) To show some of the problems AV-test has faced in the past and present with regards to sorting samples
2) To give some vendors the samples beforehand so as to reduce his workload later

Out of these scenario 1 is more likely and yes it is quite possible he may have showed older samples, because since these were used only to showcase problems, old or new doesn't really matter. The samples might still be important for companies which are new, i.e. for example VBA32.

Of course, if the reason is number 2, then I have explained this in my previous posts

 

Much credit goes to Andreas Marx for his sharing of samples with the AV companies who didn't attend, this is a true testament of his willingness to help out the end users in their protection level(s).

As for sorting and adding/sorting 15gb worth of signatures in 3 days for an unannounced test, sorry no way.
Maybe md5/crc32 detection but no "real" detections.

Correct me if I am wrong Inspector/IBK/Stephan/AV experts in general

 

The results seem inconsistent for NOD32 between the PC World Test
(http://www.pcworld.com/article/id,130869/article.html) and the most recent test (http://www.sunbelt-software.com/ihs/alex/marx/detections_2007q2.htm)

For example, Backdoors (95% (April) / 86.64% (May) ;
Bots (94/93.72) (the same) ; Trojans (89%/86.76%).

Inconsistencies exist with all the AV vendors.
Both tests use very large samples sizes.

 

VBA32 is not that new company They are just new for English market.

 

For that matter, neither is Rising. But they still deeply appreciate samples no matter how old or new they are, because not being in the English market for so long they have not quite paid focus for the malware spreading around in the English speaking regions.

 

As an update to this thread, I did contact Mr.Marx about it, and he was kind enough to explain in detail a few comments about this latest test from AV-test.org. Since the comments are quite detailed by nature, I think its better to do this post by post.

1) Regarding the somewhat "strange" detection rates of Rising, NOD32 and F-Prot when compared to AV-comparatives (for example):

The statement makes good sense to me, and I agree that an AV which does well on detecting newer threats provides an effective protection for "today's malware".

More to come in following posts.

 

2) Regarding the version of AVG tested (AVG Pro or Anti-Malware)/Some comments on product selection in general

So basically, AVG's Anti-Malware edition was used, which would explain the quite good detection rates (not that AVG Pro is bad though). As you can also see, there is some interest from the magazines in the AVG AS product, so it is also tested.

Those editions and versions are tested which are requested by the magazines.

 

3) I did ask about why AV-test does not test ArcaVir and also about why there are detection rate differences between F-Prot and Command AV. You can see the comments below:

- Regarding ArcaVir:

- Regarding F-Prot and Command AV:

So basically F-Prot will get better over time, but I am wondering why Authentium has not upgraded to the 6.x engine.

 

4) I did also ask about whether AV-test would test Virus Chaser (its anyway like Command AV in relation to Dr.Web because it has its own definitions files along with Dr.Web's database). But the comments probably will apply to test any particular AV in general, as long as its more than just a clone.

 

Mr.Marx also kindly took the time to address some of the other concerns that were being displayed/showed/posted here on Wilders.

 

And as the last paragraph of the email Andreas also wrote this

And with this, I end my long series of posts

I would like to thank Andreas Marx for taking the time to explain all this in detail, it is very highly appreciated. I hope these comments help people to understand this latest test a bit better.

Oh BTW, Mr.Marx has already seen this thread

As you can see, the tests are still very darn reliable, IMO they're as reliable as it gets.

 

Thanks, Firecat for obtaining this information . Very interesting reading.

Let's hope these emails are not pulled

 

Firecat, thank you for taking the time to contact Andreas Marx and posting the Q&A session. IMO it would have been nice to see the results of this test PRIOR to him giving out the 16+GB worth of samples. In other words some companies had 1+ week to add the samples to their signatures before the test started.

 

I do have written permission from Mr.Marx to publish those comments, so I think the comments won't be pulled.

@EliteKiller: Mr.Marx already said that the 16GB worth of samples was already uploaded on their server at May 6 and all AV companies were informed of it shortly thereafter. The workshop was the "second chance" and the email sent on May 23 was a "last chance" email to inform vendors that the samples will be deleted soon, so its best to get them now.

 

Firecat, I definitely understand the part about offering the samples before the review, which was the basis of my last post. All I was implying is that it would have been nice to see the review before the AV companies had a chance to add the 16GB worth of samples to their databases. IMHO it's no different than a professor giving the class all of the answers to the test a week or two in advance. This explains why so many had extremely high detection rates. Then again this also shows us which companies are slow to add samples. Does AV-Comparatives release their 500K samples to vendors and perform the review afterwards?

 

no. .

 

IBK, thanks for the swift reply. I edited my previous post to include additional information right before you replied. What are your thoughts on AV-Test releasing 16GB worth samples in advance and performing the review afterwards?

 

I am not going to comment it. I do not agree with some other things, but different peoples have different opinions - it does not necessarly mean that one of the two peoples opinions are wrong.

 

Hardly most Canadian beer is overrated unless it's Unibroue.. their beers are awesome! American beer is more than Anheuiser Busch, Miller and Coors as well you know. American microbrews are near the best.. only topped by Belgian Ales

Now then interesting results indeed. Avira has become quite impressive. It seems to be leading the way in detection with most tests and I applaud them for that.

I'm fine with Kaspersky and don't see myself ever switching again.. but if for whatever reason my mind was ever changed, Antivir would no doubt be the one I would try.

 

For that matter, there is nothing to prove that the 16GB collections contained only the very same samples that were used in the test. It is important to note that Marx said the collections contained "more important samples, which are already available from other sources to the AV companies".

 

True, but there is no denying that 16GB is a enormous amount of samples, and the companies that were able to implement the "new" samples into their databases prior to the testing had an advantage. Why not release the samples after the tests are completed and finalized?

 

Those companies that were able to add them prior to the test might simply be better in adding new signatures. What's wrong with that? It's not that some companies got the samples and others didn't, everyone had equal chances. By the way, it should not be the task of antivirus product testers to supply samples to the vendors. I would expect that the vendors are able to add threats themselves. You can see who's best in doing so by looking at the results. And then arguing about whether some samples are maybe less important and you will hardly encounter them .. even if there is only a slight chance of getting infected by whatever action I take, I want to be protected by my AV, that's what I have it for. I find it alarming that NOD32 which I used for years now couldn't detect some 70000 samples of threats that someone was able to spot around on the internet during the last 12 month. I like Nod32 for many things, but I'm using Avira for the moment and hope that ESET will react to the recent tests (AV-comparative test wasn't that encouraging either).