MRG Flash Tests 2011

You are not understanding what I am saying.

A sample is either current (you can be affected by it in the real world) or it is legacy (only exists in sample packs for testing).

If you collect and then test samples you are predominantly benchmarking how well you play catch up. The longer the delay the more invalid the test.

If you test samples from live sources you are benchmarking prevention ONLY and negating 'after it does not matters' definitions.

Think of it this way. If old samples were valid to test and gen/heu detections were so great then why do you need to update several times a day?

 

OK, legacy in IT has a different meaning, than your legacy AV definition.

I don't use AV realtime, see my sig (Safe Admin setup)

 

no, it means pretty much the same thing. legacy threats are older threats.

 

If by older you mean there are no longer any active infections vectors regardless of OS or installed software then yes, that is what I mean by legacy. The actual age does not matter, I have many sources over a year old that are still active while some sources I collected today will be dead within hours.

 

This is good to learn, as I seem to be hearing so much these days about older malware not being a current threat. It almost sounds as though old malware could walk in uncontested because everyone is looking for zero day stuff.

 

I've seen malware at least a year old and the sites are still up/ distributing. It's kinda funny to see that since it's fairly rare (most are down in hours.)

The reason zero-day stuff is dangerous is because we haven't found it. Once we find it isn't not dangerous. Therefor the old stuff isn't dangerous =p

 

As long as old definitions are not discarded because of their age.
Somehow I have gotten this impression.

 

Oh, possibly. But I think if they're discarded it's because they're sure they're long dead.

But I'd never rely on a blacklist anyway.

 

Yes but legacy systems are still operational IT systems in the real world

While I get the impression that nosirrah's interpretation of legacy threats means non existent in the real world

 

Definitions are certainly culled by Vendors, the time limit varying. A good part of this is due to customers screaming that the software is "bloated". On the other hand say a piece of malware hasn't made its appearance in 5 years and has been dropped by everyone. If it shows up tomorrow it won't be old anymore, but now would be considered zero day.

The circle of malware life.

 

Yes! That's precisely what I am talking about when I say, "It almost sounds as though old malware could walk in uncontested because everyone is looking for zero day stuff." Vendors tell us that they have to focus on that which is hitting the masses. That makes sense, but if malware for which definitions were culled decides to reappear, then it's time to shake hands with an old nemesis.

 

Unless it's a new variant the old stuff (individual signatures) would be replaced by generic signatures to cover a particular family of malware I would think.

 

I believe Kaspersky keeps old definitions, especially for detecting ancient boot/DOS viruses and Trojan downloaders if this 2006 blog post is anything to go by.

 

Good find, TonyW.
I quote from the blog post...

I'd like to know what other AV vendors have to say in response to this basic question. KAV made their choice clear... 5 years ago. Others (who and how many?), like cruelsister says, cull their old definitions "due to customers screaming that the software is bloated".

 

MRG is changing flash test.

http://malwareresearchgroup.com/2011/08/17/mrg-effitas-flash-tests-–-update/

 

Thanks for posting the news, tgell.

A very good test is about to get even better.

 

Good news and bring on the tests!!!

 

Great news indeed. We can't get enough tests!

 

+1! I never get bored of them

 

I'm sure you don't Noobie. BTW where is your backup solution my friend

 

Thanks for posting.
"The flash tests have become quite popular among users and some vendors, so in an effort to increase their relevance, we increased the number of samples used from one to four. As of the 29th of August, we will be introducing significant changes to the tests to further increase their validity."

I thought MRG already did multiple sample tests?

"To help give greater statistical relevance, we will include a static component to the flash tests. Twice each month, we will test using 100,000 malicious samples which are less than 72 hours old. Whilst static testing does not always assess efficacy as accurately as dynamic, it remains a convenient way to get a loose indication of performance against a large number of samples."

Nice to have some comparison with the AV-Comparatives on-demand scores.
Bi-weekly a 100000 samples, less than 3 days old, way to go...

 

yeap keep up the good work

 

My backup is . . . formatting!

 

Hahaha that works I suppose