MRG Flash Tests 2011

Microsoft Security Essentials. Windows is a closed source operating system and therefor it's Microsoft and Microsoft alone who can properly enforce security, at least in an ideal world.

Though I'd probably also throw Comodo on there for a few of its functions and then disable the vast majority of what it does. And then of course typical system hardening and explain what UAC is.

Oh and Chrome, of course.

 

I am beginning to think that these results are being cooked. If I look at this spreadsheet https://docs.google.com/leaf?id=0Bx...OTU3Yzc2NDNmYjVj&sort=name&layout=list&num=50 that indicates what kind of detection was shown by the product, I noticed that NONE, NOT ONE, of the detections in the case of Norton, came from Download Insight. And for those of us that have tested DI for a long time, know that thats impossible. It does give wrong results, but to get no detection on a series of 68 test is completely impossible. Either the installation of Norton on their test setup is broken, or something in their test setup is incorrect.

Whats worse is that MRG does not post even the hashes of the samples that they tested with, so no one can verify their results.

 

They actually supply vendors with missed samples and I don't reckon seeing an invalid one in the test bed so far.

 

The ones we missed were indeed exactly what they said they were.

 

Just my thought

 

Something is very strange here. Norton was scoring around 80%. For the last three or four tests Norton is at or close to 0%. I am not a Norton user.

 

Welcome to the world of antivirus software! Where a definition update can mean the difference between 20% chance of being infected and 100% chance.

 

I disagree. Most AV-softwares today aren't traditional AVs anymore. Look at Emsisoft for instance. They're hardly failing any tests at all. It's so far the best overall AV/AM on x64 systems (DefenseWall doesn't support x64). Most vendors these days don't rely on signatures solely.

 

All AV's rely significantly on heuristics and blacklisting. Blacklisting is inherently flawed and heuristics is hardly amazing.

Yes, some suites include HIPS defense etc. and they certainly help, especially in the av-tests, but they're often loud and always paired with the bulky, bloated, inefective and inefficient scanners.

 

Would you really say that about Prevx, EAM, Malwarebytes etc? They're performing very well. I do agree that some AVs aren't even worth trying or rely on though.

 

The malcoders are not stupid, they can counter any detection tech that can be created (aside from default deny obviously) so there wont ever be any real way to avoid the 'nose to the grindstone' factor.

We constantly create 'voodoo' that crushes entire families of malware and the reaction is to just start over with a new build, you cant heu/gen your way out of that equation.

 

That is sort of my point. The malcoders are the ones ahead in the race. But many AV/AMs these days try to stay on par with them by offering HIPS/BB etc. MBAM is doing extremely good for being a more traditional AM. It'll be great to see what you come up with in the future! But then again, MBAM isn't meant to be the 'only' protection but an additional layer. I think different combination of layers is the key sentence here to maximize your protection.

 

I agree an AV is so much more than an AV these days.
And I am not talking about the "bloated" AV's.

 

I think most involved people here at Wilders have come to the same conclusion as us. Layered protection with different approaches is a must in order to be 95% safe from online threats.

 

If I remember well MBAM is focused only on malware that the AV's miss, I think that MBAM will perform quite bad in an AV-Comparative test, for example. But is a great complement for any av.

With this I want to say that probably MBAM is far to offer a great protection being alone, and they are far of being an example of a successful AM, since malware nowadays is everything.

 

Please read #712 to see what I wrote about MBAM.

"But then again, MBAM isn't meant to be the 'only' protection but an additional layer. I think different combination of layers is the key sentence here to maximize your protection."

 

IMO they are already very successful!

 

ok

 

On the flip side you can score 99% on AV comparatives and miss everything that came out in the last 24 hours.

 

Wise words.

 

As I understand it (and i could very well be wrong) the AVs have only been tested with 68 samples since January haven't they?.While many folks like to use such tests to support or rebuke Avs ,I really think these (and a lot of the you tube tests) should really be put in perspective.They all miss something and they all have their strengths and weaknesses .By the way is it fair to include defencewall?.I mean its like matousecs list of comparing outpost ,and comodo with looknstop isnt it?.Actually the only pattern I see emerging from these and other tests ,is dump all the avs and just use defencewall (when will there be a 64 bit?) .But if we all used just defencewall and an image ,where would the fun be?

 

They are included to see if someone has come up with a HIPS exploit. When they tested malware with a valid cert some of these apps allowed it to run and thus failed.

Someone correct me if I am wrong but I think they declined to be included due to a disagreement about procedure.

The only issue I have with their testing is the scale which is just a side effect of limited resources. Id like to see an ongoing test were 50 new malware sources are tested a day and aggregated over 3 month periods of time and published once a quarter. It should be virtually impossible to pass that if your software is not up to protecting in real time.

Even if this is not a perfect test it is only part of a group of tests that all fail on some level. I have huge issues with testing that involves collecting over a period of time and testing at the end. These tests negate 0day failures and reward 'after it no longer matters' DB bloat.

At some point resources need to come together with a more 0day testing model and then everyone that deserves to win will.

 

OK, lets look at this from a malware researchers point of view, we will use our latest Flash Test as an example. We used 4 samples of Ransom Trojans, those 4 samples come from 4 different “families”, there were about 100+ variants for each “family” ,and I must emphasize, at that time as there were many more variants being released at the time the test took place. So, we are not talking about just 4 samples here, we are talking about hundreds of samples in the window of just one hour, the results would have been the same on all the samples from each “family”. To make this as clear as possible, if we tested ,lets say, 400 samples that day (100 samples per “family”), the result would have been exactly the same.

How does this work? We discover something that we think is important, verify the samples and do the test, before the test is published, samples are being sent to vendors so that they can do their bit in providing protection for their users. Don’t look at this as just tests, this is much more, the samples that we submit are used to create better generic detection, improve behavior blockers ect. And don’t think that vendors have an easy job, quite often we have online panel discussions about some new malware, it is not easy to fine tune protection as some things are just too tricky. To quote Bruce, sometimes it comes down to plan old “voodoo”.

So the next time you see a Flash Test, think of it this way, OK 4 samples, this means 4 groups of samples, this means there must must be hundreds of these samples, you can also add this “MRG guys must have sent these samples to the vendors so I should be protected by the time I finish reading the results”

Regards,
Sveta

 

i dont think thats entirely true. while having 0-day protection is important, id say more often than not people come into contact with malware that is slightly older rather than a brand new strain thats never been seen.

 

Hi Sveta thanks for the info.

I have to ask if you are going to test more frequently anytime soon like you did before? And when can we expect the next test to be published?

Keep up the good work!