Testing Heuristics Detection in a Real-World Scenario

Here is a recent article for your perusal.

 

Sounds like people from Eset tries to justify themselves for not participating to the latest av-test.org retrospective detection test. Maybe they fear that they would not have had the 1rst place, since they put a lot of marketing effort in promoting NOD32 heuristics ( a bad result would have been almost as bad as missing a VB100 for them).

Of course, the figures given by this kind of test (<40%) are probably far from what a good heuristic engine can achieve in "real world". Mainly because the time elapsed between the latest engine update and the first appearance of the malware is much longuer than what it would be in a standard "real world" use.

But still, in "real world", very few products allow to update signatures and heuristics separately. And in some products (F-Prot) there are very close links between both. And scanning a collection of already-known worms with signature scan disabled proves absolutely nothing.

Also, it is quite hard to establish accurately the time when a malware have been discovered for the first time. Of course that it would be good to scan every new malware with a scanner that is just 12 hours older than the malware itself (did I miss something or is this the author's proposal), but I think this kind of test would require a lot of work, much more than the so-called "retrospective tests". And somebody has to pay for that work...

To sum up, I somehow agree with his conception of what would an ideal proactive test. But I'm afraid this "real world" testing does not complies with real world constraints...

 

It would have been a courtesy if you had mentioned that your link would precipitate a .pdf download. Yes, I recognize that I should have looked before I leaped but I sometimes forget that there are people around who, sad to say, do this sort of thing.

The referenced article is a self-serving apologetic, under the guise of technical discussion.

I found your post enlightening, tweakie. AV tests which provide reasonably full & clear disclosure of their test bed and testing methodology can provide useful indicators of the quality of the programs being measured.

As for heuristics, DrWeb remains the king...
Heur today, guam to maui.

 

And NOD32 is one of them.