AV-Comparatives Retrospective / Proactive Test May 2011 released!

You might not be aware that we are already providing those new type of tests. I do not understand why you think that testers are doing rationalizations or give excuses. In fact you are asking for "rationalization" by asking to drop one type of test which some users are interested in.

 

ok, i agree with you on some points. so bottom line is that we need to have net for overall accuracy of testing ( as said indirectly by you ).

 

Andreas, I know that Dynamic tests won't go away.
I simply stated my preference for Dynamic tests over the Retrospective ones.
And I am Not the Only one who has this kind of preference...

 

Once there was a time, where it was very easy to see which AV's were
strong in heuristics (active/passive etc) and which were not.
Important to give you an idea how the protection of your AV was against
new and unknown threats.
If a new virus came out 1 AV did'nt have to do anything, because it's behaviour was already in build in in the heuristic filtering.
An other AV needed to update its signatures to be able to catch the new Virus, this could take weeks or even months in the worst case.
But now with CLOUD it is no longer fair, to compare products this way.
It will give a very bad, unreal comparisation.
HitmanPro was the first or one of the first that used the cloud to test files
against at least 5 antivirus engines and their own software.
If there is no cloud connection the product is not the product anymore.
All others are now doing or going to do simular things.
Sadly for the Retrospective/Proactive Test i have always appriciated that much. But those days are over now.
These test have no value for me anymore.

But most important is that i think al the 30 biggest (most used) Antimalware / Antivirus software should be tested
EVEN if the company/maker likes it or not.

And it would be great if there was a AMTSO certification, as long as there is no certification, for me as a customer the AMTSO still has real no value.

So my opinion: Drop the Retrospective test (although i feel a bit sad about that) and test 10 extra AV/AM products although they did not pay for it.

I hate it when famous AV/AM can't be compared by users
by reading these tests. If they are not tested here, they will claim and refer to all kinds of test results of the testers nobody here really likes to see.

 

First, we don't have null or poor detection rates. We actually did pretty well in this test last year when we participated. To you second point, most customers don't want to waste time, disk space, and memory scanning for every threat that has existed from the beginning of time. They prefer we stop what is coming now. That is where we have been focusing our efforts.

Static tests are the least relevant of the tests performed for determining if you will actually be protected when a threat hits your box. Freezing the definitions and disconnecting the cloud makes it even less relevant. Which are you more interested in knowing: how well your product does with disconnected internet and frozen defs or a live system being tested against real threats in the way actual way they try to attack your system?

If you don't agree, try this: block your security product's ability to access the cloud and disable its update capability; then disable any behavioral protection it has. Now you are ready to surf the internet in the same state as this test attempts to verify. Would you do that? I don't think so. So why would you want to base your evaluations on a test that tests that way?

 

What was last year may not be anymore the case now.

Most users want to be protected at any time, not only when certain circumstances are met. Not everyone is constantly online, not all malware comes exclusivly from the web, a cloud does not mean that it detects everything (and also not faster), and products without cloud are able to detect even more than cloud products (and with less FPs).
Anyway, its not about comparing tests. As said, we put lot of efforts and promote very much our dynamic tests (for which I would be glad if more enthusiam would be shown in the other thread or when the report comes out next month). If you want to compare tests with reality, first tests to look at would be ITW tests; but I doubt those ones will be ever discountinued. Even dynamic tests do not show the full picture of a product. There are also other product aspects, and evaluating them needs different kind of tests, to evaluate the heuristic/generic detection capabilities, the retrospective test makes sense. To evaluate impact on system, malware removal test does not make sense, but performance test does - so this is not a reason to drop the malware removal test.

 

Lol ! That really ... hurts !

 

Yes thats the truth. Whats the use testing a av on known samples when the internet is full unknown ones. AV makers are always behind the malware. Thats real world

 

I think not having or seeing the purpose of the Retrospective tests is being short sighted. They have and hopefully will continue. Old malware is still more prevalent then new.

 

Yeah, but let's be serious here, these days 99.9% of it does. Even if you're putting something on your computer offline, odds are it's something you got from the web. If it's something you're installing offline which you received offline then any malware contained on the media will be old and therefore it's the on-demand comparative which is more useful than the retrospective one.

I agree with tuatara, removing the online element is making the test irrelevant. Cloud scanning is becoming a vital part of any AV product, even more so when you're looking specifically at the heuristic element. I really don't understand AVC's thinking here, and the fact the vendors are withdrawing show they don't think much of it either.

 

exactly, and this is one reason why i love AV-C: methodology is changed and improved over time, so there are little chances that vendors adapt their products for scoring well in the next occasion

 

i never stated that disabling queries to the remote database in the cloud means disabling the [down]loading of local signatures on the client side

you said some of their product signatures are cloud dependants, so why not deliver signatures not cloud dependants? Is so difficult?

 

Terabytes of data?. That's the point of the cloud, using a lot more signatures than your computer can handle, permanently up to date.

And choosing a significative subset of those signatures that can provide good protection in order to be stored locally in your computer?. Yes, they already do this, but adding the cloud too is even better. As far as I know, your antivirus is beginig to follow that route too.

 

Looking forward to reading the Report...

 

+1 . Time to move on, no more excuses. Those tests do not reflect anymore the reality of how malware is distributed and spread nowadays.

 

all tests are done online, incl. on-demand test.
the retrospective test has to be done offline, because it does evaluate an aspect of the product which can not be tested online (and would not make sense to TEST online, but it does anyway show a real-world aspect; just think how condoms are being tested and how they are used in real-world), as it uses only NEW and therefore unknown malware which afterwards turned out to be prevalent.

 

Isn't the retrospective test a heuristics test, so surely it's the one where cloud support makes the most sense, not the least?

 

@Quitch: do you know TODAY what malware will be released TOMORROW and which one will turn out (in some weeks) that it was PREVALENT? No, and if you use cloud in some weeks, you will not test the heuristic, but mainly the blacklisting/signatures, so a product which has 0 heuristic but blacklists all hashes/files it finds after some weeks/months (or uses other product verdicts in the cloud), may look like having good proactive detection, while it does not, because it is not what will have been tested if you use the cloud.

 

Okay, thank you for that I better understand the AVC stance now, but I feel that you're in a corner and there's no good answer. If you switch on the cloud you can't revert the cloud to a state on date X, but let's be honest, AV makers now rely on the cloud as a first response to new threats. If you remove the cloud I'm not sure what you're really testing any more, but at the same time I don't see how you can test fairly with the cloud because it's no longer in the state it needs to be for a fair test.

Tricky. Alas, it doesn't make the AVC test any more useful because heuristics are no longer the only first response of a vendor.

 

True, "restrospective online" does not make sense unless you also get the cloud been backdated. But also the test has limited value since most products do rely on the cloud to determine positives/negatives

 

In other words, products heavily relying on advanced cloud heuristics not otherwise possible on a single PC get a unfair/unrealistic assessment by the test as compared to products with an offline standard heuristic. The test, as designed, does not supporting innovation in the sector...

 

Currently those who do not want to be in the retrospective are free to opt-out.
When - at some point in the future - we will stop providing retrospective tests (or provide it only sporadically and not regularly), I do not want to see users complaining that they miss the test.

Btw, in reality the cloud is not always the first in responding to new threats, that is just what the marketing wants you to believe; another tester showed in past what AV clouds can and can not, unfortunately that presentation is not publicly available. A lot of malware is not detected even by the cloud even if it is around since months, while others have added detection to it to their signatures (or detect it proactively by heuristics) much faster. Do you really believe that when I get infected by a new malware now (not yet covered by the cloud) and you run the malware this afternoon, the cloud will protect you? I would prefer not getting infected in first place, even if I am one of the first zero-weeks users (not to speak for the cases where vendors do not inform the users when their clouds are not being accessible due mantainance or network problems, or in corporations where cloud-access is not allowed and where some vendors provide local/in-house/offline clouds).

 

The cloud is not the solution to all security problems, it is simply another layer.

The problem seems to be that more and more of them are.

 

As we have good and bad offline heuristics we will for sure have good and bad cloud heuristics. The users will likely not miss the test if this is replaced/complemented by a compareable cloud testing.

May be we should invest some research and thinking on how to properly measure the perfomance of this 'new' security layer to distinguish between advanced security "dressing" and real improvement of user security.

 

Yes, but mainly because it is not good for their marketing. Practically all who scored low last year (or scored high last year but would score low this year) opted-out.
The same may happen in some years with dynamic tests, when some vendors will see that they are not as good as the other vendors or are at the lower end.

You are very right in saying that the cloud is one additional layer and should not be the only layer.

Also removal tests would have to be discontinued if same "logic" would be applied, as one could argue that the malware would have been blocked beforehands and there is no need to remove the malware (and we use only malware which can be detected, as undetected malware can not be removed anyway).