AV-Comparatives Retrospective / Proactive Test May 2011 released!

I don't like the fact that "cloud" has become such a marketing word. While it can certainly provide greater protection and performance when implemented in a good way, the presence of a cloud itself doesn't imply better protection and performance.

In the case of Norton, I find it more important the capabilities of the Insight and SONAR components are not included in the retrospective tests. Those two components are in my eyes the best part of Norton. (Although you could argue they don't provide detection, they just don't let something execute, and thereby those two components could provide some false positives.)
It is obvious that Norton became a top product again after introducing those components.

Norton's:

• signatures are alright, but certainly not industries best, especially considering the amount of people and money available;
• heuristics do plainly suck, cloud or no cloud;
• SONAR (and the use of File Insight data by SONAR) are obviously the components which are most important in providing proactive protection.

 

Exactly the way I feel, and I want the tests to stay a part of AV-Comparatives.

I think that some vendors are relying too much on the cloud now and not their core engine. Where is the innovation? Build a better mouse trap.

 

+1

Agree with you

 

If you look at the speed by which the AV/AM products
are implementing cloud features now, i expect that none of them
will pay for Retrospective tests within a year, and some perhaps 2.

The problem starting now is, that since more and more brands are not willing to pay any longer to have their product tested this way, for obvious reasons.
The results of the comparatives will no longer give you
any real clue on the performance of the top 30 AV/AM products.

And although it is clear that the cloud gives AV/AM product more and better possibilities (scanning with multiple engines to prevent FP's , pushing less signatures, more processing power etc etc.
The lets say ..for example ..3 only products without cloud features build in, will be the best in these tests, altough the might be the worst in real usage.

This will give end users perhaps true but very poor information,
and most certainly confusing, this will not set them on the right track.

I bet that Retrospecitve tests will not get AMTSO certified in the future

Please check which AV's are using Cloud features now, or in the 2012 versions ..

 

Good Morning ! I agree with your observation...once the Pseudo Marketing claims have been dispensed with...Cloud Protection is in my eyes still at a Beta and Refinement Level. What with sophisticated hacking attempts lately, I prefer to stay Earthbound. Sincerely...Securon

 

Imagine having the security product "xyz Internet Security" which combines the following components available on the market:

• ESET's signatures, generic signatures (static/file-based heuristics) and emulation (dynamic/runtime-based heuristics);
• Norton's SONAR and Insight network;
• and, Kaspersky's firewall and application control.

Then we could close this whole forum .

By the way, I want to apologise for slightly going offtopic with this post .

 

I am sure , no company would want that
after all its business too

 

It will never happen and forum will remained opened for discussions

 

Or just install DefenseWall and forget it.

 

youre considering a dynamic test as "a live system being tested against real threats, so in your opinion, new unknown samples in the retrospective test are not real threats to users?
Are the samples in the retrospective test old threats not coming now for you?

 

If a test freezing your local signatures is not representing adequately the real scenario, why you agree to participate in the on demand tests?

I cannot understand why you are using that answer, when to my eyes is contradictory

 

If its a extra layer why does their detection rate drop so drastically when not online?

 

+1 exactly, the cloud must be seen as a tool reducing the reaction time for the vendor, not for the client

the clients are indirectly cloud improved by mean of the vendor signatures releasing

 

Since some time ago, on demand test are done with internet connection enabled. You are not trying to understand anything, you have a fixed idea and keep going on and on and on.

For example:

"youre considering a dynamic test as "a live system being tested against real threats, so in your opinion, new unknown samples in the retrospective test are not real threats to users?
Are the samples in the retrospective test old threats not coming now for you?"

I don't know if you got something against Norton but it seems to me that you are only trying to create confusion. Those samples are no real threats because users got up to date products with all their components on, no crippled versions with ancient data bases.

 

afaik, the samples and the signatures are no more than a week of difference

i never said or named a vendor, did i? Many vendors decided not to participate in this test, because in this test cloud reputation (a reactive method) is not considered here
if you see in other security forums this problem is extended to the av industry, and so here the need of testing standards

 

IMO, this is an unnecessarily petulant reply to Mark's scholarly, well-balanced post. I am disappointed that a professional tester feels it appropriate to resort to put-downs.

 

I do not see what I wrote wrong, but maybe for native speakers there is something wrong, sorry for that. I wrote "may" - the public could know that this is the case if the result would be made public, which will not happen because we respect/offered the opt-out.

 

Can't AV-C test real zero-days with internet connection for the Retrospective Tests?

 

It is true that computers nowadays are interesting when connected to the Internet, and most people used them connected. There are however situations whereby a connection is not available for all sorts of reasons (wireless signal too weak, network security keys unavailable, cable connection too far to reach, etc). I'm obviously referring to mobile situations using notebooks.

In the last 5 years in my activity, most detection of malware was from USB flash drives and CDs of data (more than 100 instances) with my machines offline. In the same period of time there were perhaps 2-3 instances of detection when connected to the Internet. In 5 years time, I think connectivity to the Internet will be easily available everywhere in big cities, currently I'd like to rely on a scanner that can do the job even offline.

 

No, that is impossible (see post #118 ). The retrospective test has to be done offline by design, otherwise it does not measure what it intends to measure.

 

if you enable the internet connection for accessing the reputation of files, then such test would not be a retrospective test!

 

Unfortunately, such a Product does Not exist, and Wilders will have a job to do...

 

I agree norton need to improve their heuristics and signatures.
if you lose connection to the sonar /insight network you lose alot of protection. i have also noticed quite alot of fp's with sonar /insight.

symantec also seem to take ages to sort out fp's and submitted samples at least in my experience.

 

I agree that a cloud based AV needs some HIPS/Behavioral components.

Webroot Secure Anywhere/PrevX4 beta has all the ingredients it takes, just request for participation in the closed beta and you will see a lot of area's where the risk of not getting on-line is tackled, to list a few

Heuristics
- has a special off-line setting (high by default)

Behavioral shield
- assess intent of new programs before execution + enable advanced behavioral analysis to deal with complex threats
- warn (or auto decide based on other setting) when untrusted program attemps low level changes when offline

Core System shield
- assess system modifications before they are allowed offline
- prevent untrusted programs from modifying kernel & system process memory

Firewall
- prevent outbound traffic after an infection has been found

So the future is now Matthijs, you really should request participation, since Webroot/PrevX comes very close to your best of breed wish list.

 

At Symantec our layers work together. So testing the layers independently will give you a different result (and not as good a result) as testing them together. This is why Symantec advocates the Real World tests for assessing product performance against malware. For example, our software treats an executable created by Internet Explorer differently than it does should it simply encounter it already present (Download Insight). That is the result of a number of our layers working together. And if not blocked at download, when it first runs it subjected to a higher level of suspicion by our behavioral engine. Test either layer by itself and your results may vary.

There are certain edge cases that can be tested with specialized tests, but those tests should clearly explain to the user what the edge case they are testing is (for example, thumb drives on a disconnected network) so that user can assess whether or not that edge case is really important to them or not.