AV-Comparatives Retrospective / Proactive Test May 2011 released!

I dont agree the idea of one company excluding from the tests. it means they are intentionally avoiding to be tested for their weak points .

the tests should be done for whole bunch of Av who agree to participate in AV-C for that year .

 

yeah, because monkey-B is still super prevalent. better be careful with those 3.5" floppy disks.

back on topic: if a company doesn't want to have its product(s) included in a specific test, then they shouldn't have to have it included. they're the ones who have to deal with the stigma and curious people wondering why they didn't want it included.

 

my rig don't make the minimum requirements that they have for the modern day malware so its safe. I remember my first ever virus die hard 2000 on my old 286. File virus I think it was macafee boot stiffly to it out nicely.

Back then norton was the kewl thing to have. Norton commander and norton antivirus. Got a update every 2 or 3 weeks which was about 5 floppies lol

 

Ronjor wrote :

That is a step in the good way Ronjor.

It would be great if there tests like AV-Comparatives and AV-test.org were "AMTSO Certified" making it clear that al members certify the test.In that case, it would be shamefull not to have their own product tested.That way, the public would.. , just like the members wish,
immediately know, which test are certified and reliable according the
AMTSO members.And i guess that would be the fastes way to remove other unreliable and often very bad tests from the field.

Finaly , i have never said that the tests of AV-Comparatives were not good.
It is clear that most companies are proud to have AV-C. positive test results and logos on their websites.
But it is time that the general public gets to know how to value these tests.
In the end that will be better for them and the products.

Hope this positive discussion will help as well



@AMTSO members: so according tuatara: AMTSO CERTIFIED TESTS must be the agenda for the next meeting

 

There is a very simple reason why Symantec does not participate in this retrospective test: the cloud. Many of our generic heuristics -- the very ones this test aims to exercise -- require the cloud to work correctly. By disabling the internet you, in effect, disable our most generic signatures.

The way our generic detections work is to start with a detection on the file, and then to query our cloud. In the event we cannot reach the cloud, we have two choices: block or allow. For our more targeted generics, such as Suspicious.Vundo, we use the cloud to limit False Positives, so if we cannot reach the cloud we will block. But we have another set of looser heuristics, such as Suspicious.Cloud, which would exhibit too large an FP rate if we just allowed them to block, so we require the cloud to confirm. If we cannot reach the cloud, we cannot confirm, so we suppress the detection. This effectively gives us "defs in the cloud", since a relatively small number of signatures on the user's machine can result in millions of detections in the cloud. McAfee's Artemis works in a similar way, using a loose heuristic on the user's machine to drive a lookup in the cloud.

So there's the problem, if you disconnect the internet, you have dramatically impacted the effectiveness of one of our key technologies. This is true for all static tests, not just retrospectives. Static tests are dubious at best, since they only exercise a fraction of the technologies we use to protect our customers. Static tests without internet really don't give you accurate results (when was the last time you were infected while disconnected?). Isn't the point of relevant tests to accurately reflect what an average user would experience?

 

so if I place a nicley packed package onto a computer in the cloud enviroment and set the date to 2038 which means they loose internet connectivity so i can unpack and will have a better succes rate?

 

In all likelihood your initial package would get taken out on download. Additionally, to do anything useful you are going to have to have internet too. And when we do, and if you are still around, we will take you out then.

 

The retrospective tests shows you how good are the products in detecting malware proactively
In other words, how good are their heuristics capabilities for detecting unknown malware, and note that the sample set is containing fresh unknown malware from about 1 week, so this reflex how fast are the vendors for developing this heuristics, too. That is why a vendor can detect for example 70% of malware in the same test-set but they required 6 months for doing that!!!

Other types of heuristic analyzers, such as behaviour blockers, they require that you run the application for the analysis begin, so the vendors takes more time for discovering that applications, especially for rarely used applications and a slow reaction means lower detection rates

 

To add to what Mark has already said..

There are many ways by which "new" malware can be detected proactively. Two of the most commonly discussed are static or file-based heuristics and dynamic or runtime-based heuristics. These retrospective tests only exercise the former, and hence the results are clearly give no indication of whether that user would or wouldn't have been protected had they run into that piece of malware in the real-world.

 

Hi Mark,

what you stated in not very accurate (i.e. correct but potentially misleading due some missing information), as you know that I asked if Symantec would take part if we would allow to not surpress (to be exact enable the non-surpression by using a special dat file) the cloud heuristic detections to get all you generic cloud heuristic detections. Due that, I disagree with your post. Even Panda Cloud which is considered to rely a lot on the cloud can manage it to have good proactive detection rates while being offline and not depending on the cloud (not to speak about products which do not need a cloud at all and have anyway good heuristics/low FPs).
Users want to know also how good/bad "parts" of a product are, forcing users to read only the WPDT results does not work. I can understand that this would be preferable for Symantec due your good results there (and for us all due the high work we spend on it and the aim to provide one further test scenario), but the reality is that currently most users still want to see additionally also other aspects of the products and various types of tests.
You may see some few users here in the forum expressing their wish to see also such tests, I get more emails asking for them and the survey also showed that a majority still wants them (where you said that majority of users choice is wrong - sorry, but right/wrong is not what a survey reveals, it just shows what users voted for; e.g. I may disagree that 90% of a population voted for a political candidate which I do not like, but I have to accept it and can not just say "they are wrong"). I may change opinion when vendors will write on their box "requires an active internet connection to provide security". Btw, if users are at risk only while online, there would be no need to have constantly an AV running in the background while being offline and causing impact on performance...

@users on Wilders: please tell us your reasons (here in the forum, NOT by email) why you want to continue to see also retrospective tests being provided.

regards,
andreas

 

ESET version 5 has new feature called "Cloud Reputation".Does it mean ESET 5 is not qualify to be in the Retrospective Test once they officially released ESET version 5?

 

I think ESET's cloud (ThreatSense.Net) is mainly focused on improving reaction time and reducing false positives at this moment. At this moment they are not really using reputation data (prevalence and age) to add detections. This test is focused on, like earlier said, the static or file-based heuristics (generic signatures in most cases) and dynamic or runtime-based heuristics (ESET names them advanced heuristics) and those are all offline. So I think it will take more time for ESET's cloud to get mature and visibly provide better detection, for now it is just under the bonnet.

Please correct me if I am wrong.

 

thx IBK , for clear info and details about test and why norton did not participate.

I asked in their forum and they are NOT answering it .

 

Dynamic tests are the ones I really like to see; not the Retrospective ones.

 

Dynamic tests for me too. Seeing how the best performers hardly reach 60% doesn't give me confidence about zero day protection. I prefer the one that has the fastest response against new threats than the one that catches more unknown threats, if 60% is the best that it can do.

 

@Mr.PC & others: that is not the question. You will continue to see us providing dynamic tests, you do not have to fear to loose them. You can see them here: http://www.av-comparatives.org/en/comparativesreviews/dynamic-tests

 

no, if you see all retrospective test back from year 2004(heuristics) such vendor always has a detection rate higher than 50% for unknown malware and a low FP rate, and not requiring the cloud for FP suppression

 

afaik, the dynamic test is not focusing in zero day threats, although is possible that they are also present in the URL pool, in addition, due to that the case numbers is very low compared to the numbers of infected sites

probably there are more reasons that justify the importance of retrospective tests

 

Yes, that's why I made the distintion between "new threats" and "unknown threats".

 

Yes, I think that's understood. We were only expresing a preference, all the main test types are interesting. For example, I've noticed that the antiviruses that do better in the dynamic tests (Norton and F-Secure, for example), are bad or mediocre in the restrospective ones. I guess it means that they have similar approaches.

 

I agree with the explanation given by MarkKennedy. Full or partial Cloud-based AVs is the *new wave*. Testing organizations must accommodate that fact & modify their testing procedures accordingly. I hope to see better testing techniques -- NOT rationalizations & excuses by testers as to why they used antiquated test methods.

 

Yep the cloud is here ready or not.

 

+1

 

Nevis, I don't know why you did not get an answer in our forum, but it would be the same. Andreas, even if we give you specialized definitions to attempt to address the lack of cloud, the bottom line is it will not be an accurate view of how our product would perform on the test. It might make us look better, it might not, but it would not be accurate. And this is before we get to the point that new malware does not magically appear on machines, it must get created there. And we have many technologies which address the threats when they do appear. These are not tested in this type of static test. When AV consisted only of definitions and there was no cloud, this was an excellent test. However, the technologies in the products have changed and the test is no longer relevant and should be replaced by better tests (which you are doing), whether your readers understand that or not.

 

An offline cloud is not an excuse for poor or null detection rates
Why the mistery, is so difficult deliver the signatures locally? or malware authors are faster than what they are... and so the need of putting the signatures in the cloud is big?