AV-Comparatives Retrospective / Proactive Test May 2011 released!

You're right, I missed that part. Thanks for correcting me.

But that still doesn't take into consideration cloud-based heuristics.

 

True. That's by design.

 

I think this is turning into a flame war against IBK

IMO retrospective tests are great to measure performance, probably not the best "Real World" case/scenario but it does measures how a company focuses on signatures and their engine, after all, signatures are still their main weapon right? . . . Other technologies are just a complementary layer but heuristics and signatures are the main artillery.

 

+1

 

Wrong. And becoming increasingly wronger. Signatures are by their nature reactive. Our other technologies are proactive (which is one of the frustrations of the retrospective test, since it is supposed to measure proactiveness). We have been de-emphasizing signatures in favor of our other layers. Our proactive layers have been blocking more threats on our users' machines than signatures do. This is like saying that seat belts are still the main safety feature of a car. That the anti-lock brakes, crumple zones, and air bags only augment the seat belts. But that would not be correct.

Also, this has nothing to do with Andreas. It has only to do with a particular test that Andreas runs. No one that I am aware of has ever said Andreas runs the test wrong, just that Andreas runs the wrong test.

 

Current multi-layer approaches are great - but not to many layers should depend on an active internet connection.
In case of 'cloud seeding' it's not to bad, to rely on old-school (reactive) client-side signatures.

 

what about client-side heuristics?

 

Also nice, of course.
Reactive signatures should have only been an example.
Nowadays a scenario, in which it is not possible to access an av vendor's cloud / servers in realtime should not be too strange. That should have been my point - I should have been more precise.

 

Retrospective tests give a rough idea about how a product performs when it is not connected to its online database. Hence it is not useless IMO. There are still people using time based internet plans where it will be costly to remain connected to the internet all the time.

 

People may not be connected at all times, I will grant you that. But the vast majority of those people are connected at the time of infection, or they would not have been infected (or the infection is also blocked by the lack of connectivity). Infection without connection is an edge case and should be treated as such.

 

Well a friend of mine was once infected by opening a file he received via removable media. He thought it was a word document when actually it was a virus executable with word document icon. So there are cases of offline infection but are very rare these days

 

Yes, there is that case. There is also the case of viewing an email attachment after you have received it.

To be clear, I am not saying these scenarios don't happen. Nor am I saying a product should not be able to operate without the cloud (our individual heuristics each carry their own logic for dealing with the lack of a cloud response).

What I am saying is that for most users neither of those are the norm. Moreover, most users are not sophisticated and do not understand the nuances of various tests. When they see products ranked on a retrospective tests they will likely equate that with those product's nominal performance against threats, and that is the wrong conclusion to make. I am also not saying that Andreas is promoting that view, only that the only way to stop that view is to stop that test (or rename it to "Edge case no-connectivity static test of signature detection layer" -- Andreas, make a note ).

 

good point.
If you are not submitting the sample to the vendor, how can they analyze the sample code by the "cloud heuristics"?

 

I think the test is well explained at large in the report.
I do not think that censoring or forcing the remove of certain tests is a good idea (I think it will lead to the opposite), instead we explain the tests (incl. explanation of what they show and what not, their limitations, etc.) and point users to our other tests (i.e. WPDT), so they are not limited in their freedom and can look and the tests and aspect that they want to look at (and according to the poll users still demand to see such tests).
On our website we prominentely show the dynamic tests (also with monthly results, etc.). We have already dropped several tests, and for the retrospective test we even gave an opt-out possibility. I think that is more than enough (for now). I am also not asking vendors to remove from one day to another certain products or toolbars from their portfolio which I do not like or because I think they are useless (and when I sometimes ask, I do not really expect it to happen, as opinions/views on certain matters may vary).
ITW tests are criticized since years (although no user really knows what are the real reasons why most products get 100% there and some new/small products usually score badly). They still exist and will probably continue to exist (with some cosmetic changes).
Furthermore nearly all other testers (at least the big ones) include in their tests/certifications such tests that are described as outdated here; so, for those who haven't understood, the talk here is in general and not limited to avc (and please keep in mind that we are providing the dynamic test on our website and - at least currently - we use more test cases than most other testers).

 

At least in another important tester case, the similar form of outdated tests were dropped in favor of more real-time tests. I'm talking about a test similar to Retrospective which was also performed without cloud (Reaction Time) and the ITW test (which has been replaced with a more relevant ITW testbed than the outdated and limited WildList.

 

I didn't know that reaction time tests were done without cloud. But they were interesting - I would have prefered if they would be still provided but with cloud.

the ITW test is still included as part of the certification afaics (but I guess they will remove them in near future as announced; as the industry average is usually 100% for the included products, it is indeed useless to keep it in).

Similar arguments as used here against retrospective tests could be used for the behavioral tests (as also that test looks at a certain aspect of a product).

 

Andreas,

I think you own poll undercuts your position that the limitations of the retrospective tests are well understood. It seemed pretty clear that people preferred that test over the real world test. This tells me that they do not really understand what the tests are showing.

Your website calls it Retrospective/Proactive test. It is retrospective, but it is only partially proactive.

Whole Product Dynamic is better, but I think "Real World" conveys more.

You also list the retrospective higher in the list, which also serves to convey that it is more important. IMHO, Real World should top your list. Then a subcategory of Static Layer tests where you would have your on-demand and retrospective tests.

Lead with your cutting edge, not your legacy.

 

Actually we did re-arrange the order of the menu according to the poll.
I also often say that voters who voted for the opposite of me and my view are wrong and do not understand it. But it is not fair to say that; they could say you do not understand what the users are interested in or what their needs are. It is something to which users have to be educated/informed, not by forcing them to view only one type of test, but by giving them enough information to know and decide by themselves, only then they will give the deserved importance to the WPDT (and at some point in future some other tests may disappear).

We also started some months ago to rename some tests (see menu and PDFs) as suggested at last meeting (at least inside the reports, as space is limited). I also agreed to rename the WPDT to 'Whole Product "Real World" Dynamic Test', although I would have prefered to reserve the term "real world" (which I consider a buzzword) for some not yet existing tests which may come out in unforeseen future (but I had to use the term because other testers use other naming/meanings for dynamic tests).

 

Giving them what they want is not always the best route. If you had asked people in the 1890's what they want for transportation they would have told you "faster horses." Have you read the book "The Innovators Dilema"? It gives many cases of where people who were asked what they want actually wanted something completely different.

You are the expert. They look to you for guidance. I would be curious how many people who have been reading this thread currently believe that the Restrospective test is *more important* than the Real World test for determining the quality of a security product.

 

Fair enough. Good point. But in real world circumstances the products are doing worse than those tests shows to be honest. A good security product don't need this environment to give a user the best protection. They will give that protection in no matter what circumstances. They still lenient in those test by asking the vendors how they want it to be set. Wonder if every product was tested with out of the box settings.

 

I find it difficult to explain to my customers, that the retrospective test means that you update your pc , disconnected from the Internet for a week , infect the pc by removable media with only malware that came out during that week, but that you were not able to collect online via that pc.
And that the test results, doesn't say a thing on how they perform in other , near my opinion normal situation.
And that every AV has to implement cloud features to improve their products now, but that especially those new features are disabled in these test, because it always was , in history , and that for nostalic reasons, some still like to see test results of a small getting-less-important part of an AV.

Please rename this test: "Testing outdated heuristics of AVs"

 

For consumer products testers generally will use defaults, as very few users ever change them. For corporate products it gets a little murkier, since many corporations will tweak the settings (the bigger the corp, the more they will change them, often have Sales Engineers come in and help them tune the product). For those tests testers will sometimes allow the vendor to come in and set the settings.

In a recent discussion on this, we asked a tester to contact several large customers and ask them for their configuration. We felt this would give a more accurate view of how the products are configured, and would remove the possibility that a vendor would game the settings.

 

this test show the product ability to detect unknown threats, something not focused in WPDT, but complementary

 

No it doesn't. It only shows a portion of the technologies used by products to detect unknown threats.