New AV Testing Method?

http://news.yahoo.com/s/infoworld/20071005/tc_infoworld/92398

Essentially the article linked to above proposes that AV's be tested not only by scanning samples to test if the item is in the signature database, but also throwing live malware at computer protected by the product and testing if behavioral analysis protects the computer. Note, this is not the same thing as heuristic analysis of code that is not running.

I think this is a good idea. Someone else around here has got to have an opinion on this one.

 

I totally agree. I think signature based AVs will be a thing of the past in less then 2 years. Only those that have behavioral/hueristics are going to be the only way to go. I see Avira, Eset, F-Prot, Dr Web and Bitdefender as being the possible leaders in this new wave. I am sure that Kaspersky,Norton wont be far behind. It is going to be not what you know, but what you dont know, and can detect, that will be the future.

 

This is a bit strange, every antivirus will already test malware this way,
how else can they now, if malware is detected yes or now, and if so, by heuristics or by signatures?

I

 

Test, or scan is the keyword that will be gone. Now stepping out of my shell, and I shouldnt, I know for a fact that Eset, Avira and F-Prot are already headed in this direction. Gone will be the need for daily or hourly updates. What is evolving is a periodic update, or fine tuning to the engine. There will no longer be a need for scheduled scans. Nuff said, out of here.

 

I think this method can produce misleading results if, for example, someone doesnt use an AV's proactive/behavioral protection (like KAV's PDM).

 

I think it's an interesting idea.
But you can be sure that like with other tests,some fans of programs that don't do very well will question the test's integrity.

 

There won’t be a need for tests. Test what? It, or the product either succeeds or fails. There won’t be any acceptable benchmark levels to achieve. There will be feedback in forums such as this, that consumers will discuss which products fall into 1 of the 2 areas, but it will be a different day. It is coming, and it is currently being done, but some are having issues with it but they are trying. It is a total revamp of the way we currently see things. Geez, it is so easy to see if you just look. What seems so broken for some, is exactly their attempt to start achieving this.

 

In the article Symantec, Trend, Panda, Kaspersky and F-Secure were all supporting the proposed change in testing methods. Its a different list than the boutique companies mentioned above.

Right now a whold lot of AV's test well. I wonder what would happen under this proposal.

 

Signatures will never completely die off. Non-destructive malware can act very much the same as some legitimate software (especially even more so when they begin trying to specifically evade behavioral detection). This leaves it up to the user to decide if it is malware, and that could be a problem for non-technical users.

 

It's interesting to note that the mentioned vendors all incorporate some form of behavior monitoring in some form or other. Performing execution of malware would definitely help their test scores.

 

Yep, it wouldn't be fair to test "pure" AV players (Avira, ESET, Frisk, Dr.Web to name a few) together with "hybrid" products. IMO, two different tests are required:
- Tests which evaluate the quality of the scanning engine (current regular tests of AV-Comparatives, AV-Test and VB100)
- Tests which measure the "level of protection" offered. This kind of test requires a new methodology of testing and huge resources/manpower.

 

IMHO, something more efficient and proactive like that is been a long time coming, and AFAIK, all the major AV players have stood up and taken notice of the benefits + satisfaction that quality HIPS customers have found since their inception.

As far as AV's go, there can can be none any better than those who will finally impliment & employ HIPS technology in tandem with their Virus engines, and hopefully in a manner that's not so demanding on either resources or performance or both.

 

the proposed test methods are good and are applicable to all products. the results just have to be more detailed.
I announced that in future we will also do such tests (http://www.av-comparatives.org/weblog/?p=74), but as our resources are limited, it will definitly take a while until av-comparatives is ready to perform such tests in depth (e.g. along with extensive false alarm testing).

 

The problem is not to test what it detects, the problem is to find out what it shouldn't detect! Since you have to start every program in realtime to check behavior/realtime it takes more time to test even detections. BUT YOU HAVE THERE ONE BIG ADVANTAGE: Most of the Malware runs with just a single file and there aren't any major dependencies.

That is a completely different matter with clean software! Usually you need all the Dynamic Link Libraries, the required registry keys etc! If you dont have that present on your test system most of the windows applications refuse to run! (Eg with a message please reinstall the product etc) So you cannot just grab a few files as you do it with a traditional siganture/heuristic false positive test.

YOU WOULD HAVE TO INSTALL EVERY PRODUCT (to test fp's) IN A PROPER WAY TO TEST THAT! Alone that takes years for 200+ people test organisation! Otherwise it's completely useless to test that! Because what will you test? Word & Excel only? As i said before you cannot simply collect a few files and try to start them. Most of it will refuse to run because some files or registry keys are missing.

NEXT PROBLEM: How will you test different versions FROM THE SAME PRODUCT (!) aka different versions of software from the same Vendor? You need here for every version another testmachine! Since the newer version always overwrites the older one on the same machine!

Testing HIPS etc is next to impossible if you want to do that in a proper way.
Otherwise people would have already learned how many "false postives" that really produce. Aprox. 50 times more than average antivirus programs!
Even more than Fortinet !

 

The paper proposes also how to test for false alarms. yes, it is much more work, but to do it properly it must be done too.

 

I would like to think that the average user doesn't need to rely on test results to learn about the FP rates of a antimalware product. The reason the public need tests to learn about detection rates is because they rarely (relatively) come across malware; on the other hand, they work with clean files every time they use their computer, which gives them a good idea about how prone their product is to FPs.

 

Valid point and duly understood but in essence such a new technique would at least have a database from which to measure from as well as intercept potential but not neccessarily some threat, which brings AV's right back to square one again with Heuristics.

Still, HIPS is much more likely to snag the incoming signals to the system but only based on if enough of $M windows code has been accurately mapped and at Low Kernel Level i should add, which brings to light once again the same issue HIPS rely on, and that is user interaction from an informed decision, which isn't so bad since the alerted item is SUSPENDED from any actions untill AFTER (hopefully), the user is conducted an interrogation via the filename thru Google and the like, which presents yet another i'll say weakness of HIPS, an executable system file can be modified internally which leaves the user blind if intrusive code has been introduced into it. The majority of PC users simply will not afford any time to research to determine the reason for a behavioral alert.

AV's on the other hand, run the entire interior code for a match.

 

The Inspector points out very well many problems, but there is one more: the PC user may find that not all AV products of the future are of the "set it and forget it" kind. One might well have to spend at least five minutes with the help files before giving up on a good AV and settling for "that more simple" rogue AV. So there is the matter of educating the PC user, which should be a feat possible within the coming years.

Dave

 

If *THAT* would be possible it would already be done! Trust me, more than half of the users simply DONT WANT TE BE EDUCATED in regards to security matters. They even admit to risk infection before they spend time in learning (and understanding!) that.

To give u a "other" example: How many woman do you know who take their car regulary to the garage for inspection / checkup WITHOUT HAVING A SERIOUS PROBLEM? It's just a "drive from a to b" gadget for most of them. (Except the ones which are driving real expensive cars, but they most likely have anyway somebody who takes care of that)

The real problem is not teaching. The real problem is that you see for yourself a need in getting teached! And you cannot teach people that they have to been teached! That simply doesn't work. (Otherwise we wouldn't have the problem of so many infections since we're preaching since day one DON'T CLICK ON UNKNOWN EMAIL ATTACHMENTS. People still doing that!)

 

Unfortunetly true, alot of people don't even care if they get infected in a thousend different ways.

 

An employee here brought her Dell in for me and the SysAdmin to look at a couple of years ago complaining that the computer was running real slow. After checking it with various antispyware programs we determined that she had over 900 instances of spyware and viruses. She did not bring her discs to reinstall the OS (WinMe) so me and the SysAdmin took turns cleaning it out. It took us 1 1/2 days but we got all the crap out.

There was an antivirus, NAV 2004, installed but it was never subscribed to after the trial period. Her children seem to like to click on banners, go to sites that install stuff (like the Newgrounds game site used to do), and of course click on every attachment in email. The SysAdmin took NAV off since it wasn't doing anything like it was and installed AVG to give them some kind of protection.

 

Yeah, I know. Call me overly optimistic!

Dave