[Thread split]MRG Flash Tests 2012

Of course no Anti can be expected to get 100% on ALL malware every second of every day. But constantly achieving very high pass rates in ongoing tests such as MRG, clearly distinguishes the top performers, & as such are extremely valid & worthwhile, IMO.

 

How is it extremely valid? If they don't provide insight on the tests. Are you just taking their word for it? Or did you read the details on all these tests performed some place on their website? I looked and couldn't find such info.

I like the format they are using in showing which AV vendors are fast at adding detection for these malware samples. But without more info than just saying PASSED or FAILED it doesn't really make me trust them. Other than just taking it with a grain of salt.

 

As a number of folks on this thread have indicated, this test like many others should be taken with a grain of salt as there is no details provided on what constitutes a pass or a fail. No hashes provided and no URLs.

 

yea i saw & that's exactly why i say there is something wrong with it

 

New Flash Test for August 28, 2012
Zero Hour, 6h Later + 12h Later...
http://www.blog.mrg-effitas.com/

 

Pretty similar to the last few.

 

If you don't trust these tests then just move on but there are people that are still interested in these tests.

 

Yeah like me.

 

Bruce from Malwarebytes has done a little test to see how 0day samples are detected on Virustotal, and rechecking the samples at later times like MRG does. SAS does not detect any of these not even the first sample which is retested 2 weeks later. Bruce's test is small scale with few samples so it can't really say much about the detection of a particular product, but it does support MRG's findings about SAS. You can find the test in the Malwarebytes' forum general chat section.

 

Thanks once again for the heads up, LW.

 

One of the conclusion I can draw form this is that MRG and some AV producers are using the same honeypots/source. It seems highly unlikely that with thousands of new variants a day some products have consistently got 100% detection on a sample of them over several months (from June). Of course if we knew the exact detection type (signature, heuristics, etc.) we could make a better guess if this is really the case. But we don't, right?

 

yeah, i saw Bruce's tests and i liked it.

 

They do & have for some time

Info in these PDF's about the methodology etc used it their tests http://www.mrg-effitas.com/test-archive & http://www.mrg-effitas.com/current-tests

Also Sveta has posted on here a number of times about them. Try searching via his username.

Plus, PrevxHelp has posted on here previously that he appreciates their work etc.

 

they gave a short tip on how these tests being held.
and also results charts are available from this link

http://www.mrg-effitas.com/current-tests/flash-test-results/

 

Short tip not good enough. Need full detailed methodology and links to URLs, samples, or artifacts in general so we can do the test ourselves

 

none of existing companies do the job which u expect.. but av-c always lists all malicious domains which are included in their test.

 

No, most testers (all I know of) provide only to AV vendors their misses (URLs, files, hashes) after the test for verification.
It would be irresponsible to provide malware / malicious URLs to users, as well as being pointless (what would you do with hash? just search it on VT? make an own rogue AV which detects only those hashes?)

 

There is nothing stopping anyone from collecting URLs from all of the public lists and replicating what MRG is doing.

Think for a second what you need to replicate these tests:

1. A safe execution environment that can quickly be reset - clone VM, launch, test, close, delete and repeat.

2. A list of fresh malware URLs - these are posted on multiple sites and updated every day.


You also have to keep in mind that handing these sources and samples out after the fact goes directly against what they have set out to prove. They are attempting to show that samples even half a day old are completely irrelevant.

 

defensewall is included in these tests,but its not an Antivirus/Antimalware,so what I am interested to know is are these tests about blocking or detection.
Because I cant see defensewall detecting anything,but still says passed.
If blocking then i believe sandboxie should be tested as well,and I am certain that it will block all the malwares thrown at it.

 

The general advice being given by some AV experts is that this is not a good idea. However, say they did do this; by the time one wants to do the same test, it's likely the samples are pulled from the URLs listed, or the URLs themselves become "dead". I'm sure it's been said 0-day malware and the URLs they come from have a short time span.

As Bruce has said, the only real way to replicate this type of test is to do it yourself using the known lists, providing you have the means to refresh the system to restore it back to its condition before being "infected".

 

AV-C is living in the last decade. You think the malware authors aren't smart enough that they dont know what are the holes in various products. Sorry, but a poor excuse.

Besides there are lots of sites all over the web where you can download malware. Heck, some of them even allow you to request a specific piece of malware by hash, any people will oblige.

 

There are many tools available to capture the HTTP traffic, via a proxy. For example, Fiddler. You are correct, URLs are somewhat useless. But with the Fiddler logs I can completely re-create the attack any time I want. Post those. Or post all the artifacts that the tester gives to the AV vendors after each test. They do give them that don't they ?

 

Or they have insiders on 0-day Malware boards. Don't be shocked this would happen, I'm sure it does and has been for some time.

I think more tests the better, it's valuable information for us end users. IF a product stinks like SAS then we know not to fork over our hard earned money for it. MRG doesn't look to be a sponsored test so I would take them at their word.

 

http://www.blog.mrg-effitas.com/
9.1.12 Flash Test
Avast has really stepped it up.

 

Thanks Thankful...man, it's becoming the "witching hour" for some but as you said, Avast is stepping up!