[Thread split]MRG Flash Tests 2012

New results..........
Flash Test 13.08.2012 Zero Hour + 6 Hours Later
http://www.blog.mrg-effitas.com/

 

Thanks for the update, results are quite similar from previous tests, top performers keep showing good results while others keep trying.

 

Why are Webroot not participating in these tests? - just that i thought one of the positives of WSA was that the cloud enabled 0-day malware to be identified as malicious in the cloud and all users to be protected quicker than traditional updates would? This type of MRG test would illustrate whether that is the case?

 

I had the same question few pages before... and here you find the answer: https://www.wilderssecurity.com/showpost.php?p=2086697&postcount=114

 

Thanks. So the answer is Webroot could be in the MRG test if they wanted to...... so... they don't want to?

Just makes me slightly wary of Webroot claims that they might miss the odd 0-day malware, but due to mega fast cloud technology, their users will be protected quicker than traditional AVs due to no need for download/update of signature database. And their 'roll-back' feature.

Many of the AVs tested by MRGs are 'hybrids' anyway in that they also have cloud technology built in, often with sig database as well.

Just disappointed WSA didn't hang in there for a while longer...... so we could see the promised improvement(s).....

 

This has been discussed here.

 

Kaspersky is killing the zeroth hour. Impressive!

 

17.08.2012 - 0, 6 and 12 hours.

SUPERAntiSpyware included.

 

SUPERAntiSpyware should be renamed to FAILUREAntiSpyware ...

 

Not very impressive results from SAS.
Nice to see DW stay on top.

 

Not a very impressive performance for anyone, save Kaspersky, Emsi, and Bluepoint.

 

I presume your talking overall results?

 

Zero hour results for this specific test.

 

And once again the usual top performers.
Wonder why even after 12 hours SAS only caught 1.

 

MRG doesn't publish any info about the samples they test with.. no hashes, nothing. So its impossible for anyone to corroborate their results. For all we know they could be another antimalware.ru outfit sponsored by Kaspersky where Kaspersky magically detects everything at zero hour (which we all know is impossible).

MRG, if you want people to take you seriously, publish the samples, or at least the hashes of the samples you are testing with.

 

This is a more realistic test where web blocking also counts, MD5s wont tell you anything about malware blocked that way. Sources die too quickly to publish as well. If the MD5s and sources were to be published everyone would be complaining about vendor X not really blocking sample Y (because it was web blocked but not file blocked and a MD5 does not tell you this) and source A being dead so nothing can be verified.

Its funny that some people call this testing crap but a test where everyone does between 90% and 99% is somehow trustworthy.

I said this earlier but I will bring the point up again. Anyone with at least moderate experience can install a VM, make some duplicates, collect some live URls from the web and do this test themselves against any AV you want. If anything this testing model is more legit as you can replicate it yourself if you don't buy the results.

In reality you should not trust any test that you cannot replicate yourself.

 

Bruce,

No one is saying they should not test with live URLs. They absolutely should. But just because the URL blacklisting component of the product missed the threat, doesn't mean that the product as a whole did not prevent the infection. As their own methodology suggests, they do run the sample, assuming the product did not block the URL. WHERE IS THAT SAMPLE. If a product did not prevent the infection even though it had multiple opportunities to do so by browsing to the URL, by the PE file (if any ) being created on the disk, by running the PE file and it missed all of them, I would like to see that malicious file that they ran.

So I do think asking for the hash of the file in cases where the product(s) completely missed it, is reasonable.

Like you said, one should not trust any test one cannot replicate, and I cannot replicate this test with the information provided.

 

I've got the same idea about the MRG tests..

i think it's like this. my guess
MRG's owner = kaspersky
NSS's owner = Trend micro

 

[joking]
Not really...that would mean, that kaspersky had connections with warez sites like ssupdater.com.
Another point: Some other products (Emsisoft, DefenseWall, Bluepoint) score top at MRG for a long time, longer than KL. Why no one suspects this companies?
conclusio: not worth a discussion

and btw.: KL scored good/top in many other tests.

Beside that the valid points are:
- Are those tests transparent? No
- Is MRG trustworthy? That question anybody has to answer for him/herself.

I don't think those 95+% mass-tests have real world relevance but that says not much about their trustworthiness. At least IBK and other didn't play "sock puppet" here at wilders...

 

The tests you can trust least are AV-Test.org tests and the tests you can trust most are Dennis Technology Labs tests. Based on transparency.

 

You should also not trust posts telling you what to trust and what not. Read yourself the methodology used, assess based on your own criteria and main interest and most of all: try the products.

 

What do you mean with suspecting these companies?
I don't see any reason to suspect Emsisoft or Ilya/DefenseWall.
Core-tech used in those programs (also Bluepoint) is mainly whitelisting, imo not surprising to see different results compared to heavy blacklisting based tech.
Besides, if KL performs well for a prolonged period, as it does now, would that make KL also more suspect?
Rare but now I don't get your pov.

As the sample set used is minimal and would have to be chosen with near perfection, in order to be representative for 'all malware out there', I see the test much more as a 'responsiveness test' than detection results.

 

I have no reason to doubt any testing company as MRG, AV-Test, and Av-Comparatives all show Kaspersky performing well. I am not a user of Kaspersky.

 

I used the "joking" -Tag before, because I don't suspect any vendor. I was only referring to qakbot ("hey could be another antimalware.ru outfit sponsored by Kaspersky where Kaspersky magically detects everything at zero hour (which we all know is impossible)") and his logic.