Shadowserver Zero Day or Proactive Testing

Which method of testing yields more accurate results?

At Shadowserver I believe the AV's run up to date signatures and are graded on how well the perform at detecting new malware on a daily basis. The typical proactive test involves using signatures that are out of date by a week or more and grading detection on new malware that came in over the time period that the signatures were held constant.

I think Shadowserver has a better idea simply because we run our AV's up to date in real life.

Letting one's signatures get a week (or more) out of date is not a good practice. It might demonstrate some technical superiority, but not necessarily in the area of preventing real world threats.

What do the rest of the members around here have to say about this?

 

Without getting into the debate about what version, os etc shadowserver uses (an issue debated in the past), I think the core idea behind shadowserver and avcomparatives is different.

Avcomparatives is trying to isolate the effect of heuristics. This is why signatures is not updated for a week. It is assumed that new virus have to be detected via a heuristic techinque.

Shadowserver on the other hand is just testing for how AVs detect malware on the same day it is released to simulate a more realistic user experience. However this means that if the vendor responds fast with signatures, it may also get a good result even if heuristics is poor.

 

In theory it seems to be a more true to life approach for ratings of AV efficacy.
However, VBA32 for quite a while had been rated abysmally for no apparent reason. Evidently they had been running in demo mode (without updates) after license expiration!
For them to be running an AV with an expired license and subsequently rating it's performance is very sophomoric / unprofessional.

 

the difference is (and i think bob d sort of alluded to this) what shadowserver is doing is actually not proactive testing... it's testing against the things it finds in the wild but there's no guarantee those things are actually new (and thus that detecting them represents proactive protection)...