New AV Testing Method?

Discussion in 'other anti-virus software' started by Diver, Oct 5, 2007.

Thread Status:
Not open for further replies.
  1. Diver

    Diver Registered Member

    Joined:
    Feb 6, 2005
    Posts:
    1,444
    Location:
    Deep Underwater
    http://news.yahoo.com/s/infoworld/20071005/tc_infoworld/92398

    Essentially the article linked to above proposes that AV's be tested not only by scanning samples to test if the item is in the signature database, but also throwing live malware at computer protected by the product and testing if behavioral analysis protects the computer. Note, this is not the same thing as heuristic analysis of code that is not running.

    I think this is a good idea. Someone else around here has got to have an opinion on this one.
     
  2. trjam

    trjam Registered Member

    Joined:
    Aug 18, 2006
    Posts:
    9,057
    Location:
    North Carolina
    I totally agree. I think signature based AVs will be a thing of the past in less then 2 years. Only those that have behavioral/hueristics are going to be the only way to go. I see Avira, Eset, F-Prot, Dr Web and Bitdefender as being the possible leaders in this new wave. I am sure that Kaspersky,Norton wont be far behind. It is going to be not what you know, but what you dont know, and can detect, that will be the future.
     
  3. Joe_Jones

    Joe_Jones Registered Member

    Joined:
    Aug 31, 2007
    Posts:
    41
    This is a bit strange, every antivirus will already test malware this way,
    how else can they now, if malware is detected yes or now, and if so, by heuristics or by signatures?

    I
     
  4. trjam

    trjam Registered Member

    Joined:
    Aug 18, 2006
    Posts:
    9,057
    Location:
    North Carolina
    Test, or scan is the keyword that will be gone. Now stepping out of my shell, and I shouldnt, I know for a fact that Eset, Avira and F-Prot are already headed in this direction. Gone will be the need for daily or hourly updates. What is evolving is a periodic update, or fine tuning to the engine. There will no longer be a need for scheduled scans. Nuff said, out of here.
     
  5. WSFuser

    WSFuser Registered Member

    Joined:
    Oct 7, 2004
    Posts:
    10,632
    I think this method can produce misleading results if, for example, someone doesnt use an AV's proactive/behavioral protection (like KAV's PDM).
     
    Last edited: Oct 5, 2007
  6. the Tester

    the Tester Registered Member

    Joined:
    Jul 28, 2002
    Posts:
    2,854
    Location:
    The Gateway to the Blue Hills,WI.
    I think it's an interesting idea.
    But you can be sure that like with other tests,some fans of programs that don't do very well will question the test's integrity.:thumbd:
     
  7. trjam

    trjam Registered Member

    Joined:
    Aug 18, 2006
    Posts:
    9,057
    Location:
    North Carolina
    There won’t be a need for tests. Test what? It, or the product either succeeds or fails. There won’t be any acceptable benchmark levels to achieve. There will be feedback in forums such as this, that consumers will discuss which products fall into 1 of the 2 areas, but it will be a different day. It is coming, and it is currently being done, but some are having issues with it but they are trying. It is a total revamp of the way we currently see things. Geez, it is so easy to see if you just look. What seems so broken for some, is exactly their attempt to start achieving this.
     
  8. Diver

    Diver Registered Member

    Joined:
    Feb 6, 2005
    Posts:
    1,444
    Location:
    Deep Underwater
    In the article Symantec, Trend, Panda, Kaspersky and F-Secure were all supporting the proposed change in testing methods. Its a different list than the boutique companies mentioned above.

    Right now a whold lot of AV's test well. I wonder what would happen under this proposal.
     
  9. VikingStorm

    VikingStorm Registered Member

    Joined:
    Jun 7, 2003
    Posts:
    387
    Signatures will never completely die off. Non-destructive malware can act very much the same as some legitimate software (especially even more so when they begin trying to specifically evade behavioral detection). This leaves it up to the user to decide if it is malware, and that could be a problem for non-technical users.
     
  10. solcroft

    solcroft Registered Member

    Joined:
    Jun 1, 2006
    Posts:
    1,639
    It's interesting to note that the mentioned vendors all incorporate some form of behavior monitoring in some form or other. Performing execution of malware would definitely help their test scores.
     
  11. lucas1985

    lucas1985 Retired Moderator

    Joined:
    Nov 9, 2006
    Posts:
    4,047
    Location:
    France, May 1968
    Yep, it wouldn't be fair to test "pure" AV players (Avira, ESET, Frisk, Dr.Web to name a few) together with "hybrid" products. IMO, two different tests are required:
    - Tests which evaluate the quality of the scanning engine (current regular tests of AV-Comparatives, AV-Test and VB100)
    - Tests which measure the "level of protection" offered. This kind of test requires a new methodology of testing and huge resources/manpower.
     
  12. EASTER

    EASTER Registered Member

    Joined:
    Jul 28, 2007
    Posts:
    5,632
    Location:
    U.S.A. (South)
    IMHO, something more efficient and proactive like that is been a long time coming, and AFAIK, all the major AV players have stood up and taken notice of the benefits + satisfaction that quality HIPS customers have found since their inception.

    As far as AV's go, there can can be none any better than those who will finally impliment & employ HIPS technology in tandem with their Virus engines, and hopefully in a manner that's not so demanding on either resources or performance or both.
     
  13. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,819
    Location:
    Innsbruck (Austria)
    the proposed test methods are good and are applicable to all products. the results just have to be more detailed.
    I announced that in future we will also do such tests (http://www.av-comparatives.org/weblog/?p=74), but as our resources are limited, it will definitly take a while until av-comparatives is ready to perform such tests in depth (e.g. along with extensive false alarm testing).
     
  14. Inspector Clouseau

    Inspector Clouseau AV Expert

    Joined:
    Apr 2, 2006
    Posts:
    1,329
    Location:
    Maidenhead, UK
    The problem is not to test what it detects, the problem is to find out what it shouldn't detect! Since you have to start every program in realtime to check behavior/realtime it takes more time to test even detections. BUT YOU HAVE THERE ONE BIG ADVANTAGE: Most of the Malware runs with just a single file and there aren't any major dependencies.

    That is a completely different matter with clean software! Usually you need all the Dynamic Link Libraries, the required registry keys etc! If you dont have that present on your test system most of the windows applications refuse to run! (Eg with a message please reinstall the product etc) So you cannot just grab a few files as you do it with a traditional siganture/heuristic false positive test.

    YOU WOULD HAVE TO INSTALL EVERY PRODUCT (to test fp's) IN A PROPER WAY TO TEST THAT! Alone that takes years for 200+ people test organisation! Otherwise it's completely useless to test that! Because what will you test? Word & Excel only? As i said before you cannot simply collect a few files and try to start them. Most of it will refuse to run because some files or registry keys are missing.

    NEXT PROBLEM: How will you test different versions FROM THE SAME PRODUCT (!) aka different versions of software from the same Vendor? You need here for every version another testmachine! Since the newer version always overwrites the older one on the same machine!

    Testing HIPS etc is next to impossible if you want to do that in a proper way.
    Otherwise people would have already learned how many "false postives" that really produce. Aprox. 50 times more than average antivirus programs!
    Even more than Fortinet ! :D
     
  15. IBK

    IBK AV Expert

    Joined:
    Dec 22, 2003
    Posts:
    1,819
    Location:
    Innsbruck (Austria)
    The paper proposes also how to test for false alarms. yes, it is much more work, but to do it properly it must be done too.
     
  16. solcroft

    solcroft Registered Member

    Joined:
    Jun 1, 2006
    Posts:
    1,639
    I would like to think that the average user doesn't need to rely on test results to learn about the FP rates of a antimalware product. The reason the public need tests to learn about detection rates is because they rarely (relatively) come across malware; on the other hand, they work with clean files every time they use their computer, which gives them a good idea about how prone their product is to FPs.
     
  17. EASTER

    EASTER Registered Member

    Joined:
    Jul 28, 2007
    Posts:
    5,632
    Location:
    U.S.A. (South)
    Valid point and duly understood but in essence such a new technique would at least have a database from which to measure from as well as intercept potential but not neccessarily some threat, which brings AV's right back to square one again with Heuristics.

    Still, HIPS is much more likely to snag the incoming signals to the system but only based on if enough of $M windows code has been accurately mapped and at Low Kernel Level i should add, which brings to light once again the same issue HIPS rely on, and that is user interaction from an informed decision, which isn't so bad since the alerted item is SUSPENDED from any actions untill AFTER (hopefully), the user is conducted an interrogation via the filename thru Google and the like, which presents yet another i'll say weakness of HIPS, an executable system file can be modified internally which leaves the user blind if intrusive code has been introduced into it. The majority of PC users simply will not afford any time to research to determine the reason for a behavioral alert.

    AV's on the other hand, run the entire interior code for a match.
     
  18. dw2108

    dw2108 Registered Member

    Joined:
    Jan 24, 2006
    Posts:
    480
    The Inspector points out very well many problems, but there is one more: the PC user may find that not all AV products of the future are of the "set it and forget it" kind. One might well have to spend at least five minutes with the help files before giving up on a good AV and settling for "that more simple" rogue AV. So there is the matter of educating the PC user, which should be a feat possible within the coming years.

    Dave
     
  19. Inspector Clouseau

    Inspector Clouseau AV Expert

    Joined:
    Apr 2, 2006
    Posts:
    1,329
    Location:
    Maidenhead, UK
    If *THAT* would be possible it would already be done! Trust me, more than half of the users simply DONT WANT TE BE EDUCATED in regards to security matters. They even admit to risk infection before they spend time in learning (and understanding!) that.

    To give u a "other" example: How many woman do you know who take their car regulary to the garage for inspection / checkup WITHOUT HAVING A SERIOUS PROBLEM? It's just a "drive from a to b" gadget for most of them. (Except the ones which are driving real expensive cars, but they most likely have anyway somebody who takes care of that)

    The real problem is not teaching. The real problem is that you see for yourself a need in getting teached! And you cannot teach people that they have to been teached! That simply doesn't work. (Otherwise we wouldn't have the problem of so many infections since we're preaching since day one DON'T CLICK ON UNKNOWN EMAIL ATTACHMENTS. People still doing that!)
     
  20. GES/POR

    GES/POR Registered Member

    Joined:
    Nov 26, 2006
    Posts:
    1,490
    Location:
    Armacham
    Unfortunetly true, alot of people don't even care if they get infected in a thousend different ways.
     
  21. midway40

    midway40 Registered Member

    Joined:
    Jul 24, 2006
    Posts:
    1,257
    Location:
    SW MS, USA
    An employee here brought her Dell in for me and the SysAdmin to look at a couple of years ago complaining that the computer was running real slow. After checking it with various antispyware programs we determined that she had over 900 instances of spyware and viruses. She did not bring her discs to reinstall the OS (WinMe) so me and the SysAdmin took turns cleaning it out. It took us 1 1/2 days but we got all the crap out.

    There was an antivirus, NAV 2004, installed but it was never subscribed to after the trial period. :rolleyes: Her children seem to like to click on banners, go to sites that install stuff (like the Newgrounds game site used to do), and of course click on every attachment in email. The SysAdmin took NAV off since it wasn't doing anything like it was and installed AVG to give them some kind of protection.
     
  22. dw2108

    dw2108 Registered Member

    Joined:
    Jan 24, 2006
    Posts:
    480
    Yeah, I know. Call me overly optimistic!

    Dave
     
Loading...
Thread Status:
Not open for further replies.