Shadowserver Zero Day or Proactive Testing

Discussion in 'other anti-virus software' started by Diver, Jun 25, 2008.

Thread Status:
Not open for further replies.
  1. Diver

    Diver Registered Member

    Joined:
    Feb 6, 2005
    Posts:
    1,444
    Location:
    Deep Underwater
    Which method of testing yields more accurate results?

    At Shadowserver I believe the AV's run up to date signatures and are graded on how well the perform at detecting new malware on a daily basis. The typical proactive test involves using signatures that are out of date by a week or more and grading detection on new malware that came in over the time period that the signatures were held constant.

    I think Shadowserver has a better idea simply because we run our AV's up to date in real life.

    Letting one's signatures get a week (or more) out of date is not a good practice. It might demonstrate some technical superiority, but not necessarily in the area of preventing real world threats.

    What do the rest of the members around here have to say about this?
     
  2. jrmhng

    jrmhng Registered Member

    Joined:
    Nov 4, 2007
    Posts:
    1,268
    Location:
    Australia
    Without getting into the debate about what version, os etc shadowserver uses (an issue debated in the past), I think the core idea behind shadowserver and avcomparatives is different.

    Avcomparatives is trying to isolate the effect of heuristics. This is why signatures is not updated for a week. It is assumed that new virus have to be detected via a heuristic techinque.

    Shadowserver on the other hand is just testing for how AVs detect malware on the same day it is released to simulate a more realistic user experience. However this means that if the vendor responds fast with signatures, it may also get a good result even if heuristics is poor.
     
  3. Bob D

    Bob D Registered Member

    Joined:
    Apr 18, 2005
    Posts:
    1,150
    Location:
    Mass., USA
    In theory it seems to be a more true to life approach for ratings of AV efficacy.
    However, VBA32 for quite a while had been rated abysmally for no apparent reason. Evidently they had been running in demo mode (without updates) after license expiration!
    For them to be running an AV with an expired license and subsequently rating it's performance is very sophomoric / unprofessional.
     
  4. kwismer

    kwismer Registered Member

    Joined:
    Jan 4, 2008
    Posts:
    240
    the difference is (and i think bob d sort of alluded to this) what shadowserver is doing is actually not proactive testing... it's testing against the things it finds in the wild but there's no guarantee those things are actually new (and thus that detecting them represents proactive protection)...
     
Loading...
Thread Status:
Not open for further replies.