Interesting Emsisoft Tests

I think this is the major problem: Maybe it should be called Malwarebytes Anti-Rogueware because that's what it seems to be

 

Christian Mairoll [Emsi Software Team]

You just dont get it do you

Peoples real world experiences of MBAM(&SAS) alike in no way reflect your contrived test results.

My NDA prevents me from discussing under the hood technology or detections routine in detail used by MBAM but then again why should we tell you why the software works a damn sight better than yours at removing active malware infections during real world usage.

Ok here's a little pointer

Why create how many 100's of strings to detect these samples and thus have to parse a larger DB when using a different approach means that a lot less DB space is needed to get the job done in the real world.

While your guys are busy stringing the usual suspects,our team is off hunting new malware that evades our DB in realife testing scenario's and writing new rules to attack them...this i why we are at the top of the malware killing food chain.

So if your sincerly interested in some real world comparison on detection and removal abilities here's some pointers for tests that would reflect a more truer picture of things.

Take SAS,MBAM and a2 only.

5 common infection scenario's

1)Find a few fake scanner pages and load up the FakeAlert rogues.

2)Hit the keygen sites(Type I+II) and run ther nice free keygens

3)Fire up P2P and grab yourself a worm infection.

4)Add application at my space and grab Koobface

5)Install codec for free pr0n.

Save image of each infection then test each of the 3 softwares against the infected image(s) and then publish your results.

All i can say is i hope you like coming bottom of the class but i guarantee you wont want to be lauding the results from realworld tests such as thoes


Anyway with that can you pass a message onto your dev department as i found a real nasty side effect of your detection and removal engine when i last tested on a heavily infected machine(Keygen Type I).

You need to use whitelisting on critical system files.
Deleting system files that have been attacked by PE infectors only result in collapsing the OS and killing that install

It's ok that the system will need R&R because of the infection type but in deleting the critical files without warning that it will collapse the OS in no way gives a person chance to export(save) any pictures,media,personal documents etc

 

Hmm, looks like I found the .1% A Squared misses and the 1.4% MBAM hits.

 

lol Franklin,

To be fair the only way to compare would be to run the installer and get the realworld infection(Rogue install) then test both softwares against it.

That said the a2 tests remind of the old saying there are lies,damn lies and statistic's

 

Yes, you see that would make more seance, and a lot less people would be confused.

Now we can all love or hate a-squared, one thing is for sure it does have an excellent detection ratio, False Positives sure they have a problem with them, but so does Avira, G Data, Avast.......

From all the talk I see here, everybody seems to be concerned about removing malware from the infected system, correct me if I'm wrong but it is ten times more important to prevent infections?
If you want to be good at preventing you have to detect, if you want to be good at detecting you need to think about both present and the past as well as the future, to make a long story short most of the vendors are getting their signature databases bigger and that is very smart because all the other systems are not yet advanced and functional enough to be the first line of defense........
You can't have a good Anti-Malware product with only a few hundred thousand signatures, ask malware researchers how many new malwares are being distributed/discovered every month, way too many
High detection ratios come with the price, the price being paid are False Positives, but in most cases (if system files are not the problem), you can fix the FP's, in real circumstances it is very hard to clean a heavily infected system.

While all of you are arguing about this, there is a global pandemic of malware, it would be smart to put aside all the differences and work together for a change, only that way you can expect positive results

 

I agree, but then you can also kick AV-Comparatives tests

 

Hello,

I think it would be important to point out that, for the user, what you're saying is not that important. It would be important if, while executing said malware, MBAM would still not detect and block the threat.

Your test, and lets assume it's completely true, does not reflect 'real world scenarios'.
I'm not saying it's completely worthless, as i like reading on AV-Comparatives myself. But it does not tell the whole story, or depending on the products tested, not most of the story. Programs like Prevx for instance, or on the other extreme, MBAM, where it's developers explain why right-clicking a file to scan is not how it works best. Even BOClean, now gone.

This applies to AV's in general too, because they're not sleeping or watching other companies innovate, but to a less extent i think.

Testing isn't easy, but it's extra hard to ascertain what your "inhouse test" actually means, being a Marketing gimmick.

Don't take this as an aggressive post towards you or your company. It's my honest opinion.

 

hey guys, I don't want to be like a "pacificator" but I believe people is bashing too much in this thread: emsisoft probably shoulnd't have shared this post or shouldn't have listed a-squared, and maybe Mike shouldn't have posted it, but:

1. also other companies do the same thing:
   
   
   • why nobody insults Prevx just because they have a comparison on their homepage -http://www.prevx.com/ ?
     
     
   • eset norton kaspersky and so on have done the same thing, but I haven't seen anyone saying that they are bad companies just because they did that
   
   
2. It's clearly written that it doesn't represent an objective assessment of the detection performance of a-squared Anti-Malware, because they are samples of emsisoft
   
   
3. the test is still good to see how the other softwares perform in an on-demand scan
   
   
4. I don't understand why you accuse Mike of being unprofessional: yes, they have an alliance with emsi, but if he hadn't posted the test, probably someone else would have done it soon.. so... dunno

Now, the only thing I didn't like is that Christian wasn't that nice with MBAM, but he has a point, the homepage of MBAM isn't clear about its functionalities, limits and so on. It seems like the only thing he wanted to do with this test, was pointing out MBAM's bad performance.. That's not what you wanted to do, Christian, was it?

 

FCUKDAT, FATDCUK, ADE

A Malware Researcher should have a library of malwares, a honeypot. What would be nice for a malware researcher is to proof it, let us show us the results of such a test.

Regards Kees

 

I see nothing wrong with the test myself. The results seem to be very similar to those I have seen elsewhere. would like to have seen Prevx there as well.

 

Agree.

Use what you like, and what you feel comfortable with. If it's the free open source Moon Secure AV, so be it.

 

Right without goin too much into detail but trying to make a simplified explaination this is an incorrect assumption because of what constitutes an individual signature and how it would perform in the real world.

There are many ways to detect and attack malware with different types of detections.
The figures i'm about to quote are pulled out of the air but will hopefully give a picture of what is going on.

Take a single trojan file and you can attack it by MD5 hash hit= 1 signature for that file.

The next time you download that file it has been tweaked and although it dose the same thing it has new MD 5 value (old signature now dose'nt work)

So next step up is to string the file so you are no longer dependent on 1 file= 1 signature ratio. A string might be worth say a 1000 MD5's depending on how long it is present in the target file.

So 1 string detection signature is a lot more valid then 1 MD HASH hit signature in the DB.

Now the fun begins because even string signatures have a limited shelf life as the original trojan gets tweaked some more to evade detection.

So this is where Heuristic's come into play
A good heuristic signature with no f/p generation is worth its weight in gold.

In certain cases that 1 Heuristic signature is worth every variation of that particular trojan from past,present and future variants yet to be even distributed.

Now simple maths says that 1 signature would take the place of potentially 100 of thousands of MD 5 hits or 100's of strings hits.


I hope this explains why the size of a softwares DB(number of signatures) is not proportional to its overall detection abilities

 

I'm not sure he actually has a point. Users expect protection, i don't see a static folder scan (with malware installers sort of speak) as indicative, or as a final word on how a product will ultimately protect the users.

 

Ade, you're now presenting MBAM, like the emsisoft test - saying your product is top of the malware food chain. Not that I disagree your product is exceptional, but people from other companies that get similar results to emsisoft (Norton, Avira, G-Data, especially Avast which is signature based), could say, serve up the detailed results.

This is like a pissing contest. Each product has its own strengths.

If everyone jumps on every company test, then no company will publish any results for fear of backlash. All we'll have is a few reviewers to rely on, that's it.

 

i think a2square is an amzing product other than the many false positive which i dont mind A2 catches alot of nasties that also other misses and this is base on my own personal testing real malware that comes from danger websites(real situation)of all antimalware/antivirus signiture base shield this is the one i like(the only one)then my priority is Defensewall/malware defender

 

I have seen cases where MBAW did not detect a rogue scanner installed while A-squared did and removed it (visa versa!). I mean why are we bashing A-squared and/or MBAW/SAS while neither of them are the 'holy grail ', every product has his good and bad sides! Every test must be taken with a pinch of salt, Kingsoft/Rising for example work great in Asia, but score bad in our ' western' tests.

 

I consider indeed unprofessional his action, but if I would like to be more gentle, respecting Mike N. for his great product ( since I don't know much about him ), I would call it ... unfortunate coincidence. If I was him...I would have let Emsisoft post the test...or like you say somebody else. I understand that because of this "alliance" he monitors the actions and progress of his partner...but hey life and forum experience command a different approach...better thinking before some actions. Nothing that bad though...things like these happen.

You have missed some reading here. There is a thread about it.

I understand it and agree. But obviously others see it in a different way, which is also fair, they have theirs reasons and views...I accept it.

 

I very well know that, but very few programs have heuristics that produce almost no FP's, so it is a win lose game just a matter who is assessing the the results.

Now I see you are a Malwarebytes researcher so I would like to ask you a simple question with no bad intentions: In your honest opinion, do you believe that Malwarebytes Anti-Malware can provide the same (or higher) level of protection then ,lets say, Avira, Kaspersky, Avast or G Data?

 

You have just ask me to compare apples and orange's was that your intention

Both have their respective strong points and both have their weak points.

Personally if someone needs a blacklist scanner to do their decision making for them then both are required.

Just food for though but if AV was all dominating then there would be no need for the *cleaners* nowadays but its very obvious that is not the case.

 

So what you are saying is that Malwarebytes is a cleaner?

 

Exactly! This malware needs to be executed in order for SAS and MBAM to show their prowess.Ok maybe Emsisoft made a genuine error in lumping these two in with a load of AVs for what was essentially an on demand scan of inert malware,if that's the case they should remove them from the results table until such time as they can run an appropriate test designed to show real-world performance.People will think far higher of them if they admit their mistake.

 

The issue isn't the results that show AVs doing what they were designed to do it's the inclusion of products that shouldn't have been there.

 

No offence but I never saw anyone's PC get infected by a static folder of samples.

 

All botkillers are *cleaners* and most folks first encounter with MBAM or like software is when there pc's have been infected and they need software to come in and get rid of the mess.

Suffice to say MBAM's high clean up sucess rate is why it has gained popularity and a reputation that it has

 

I think its the description of these utilities that has caused a problem. There are no AVs any more, everyone talkes about malware and AMs! A2 is called an AM, so are SAS and MBAM etc.

I guess you are saying SAS and MBAM are different sort of AM utilities. I hear lots about MBAM in particular being designed to detect what the mainstream AMs miss and so give users a better chance of getting towards 100% coverage.

I guess SAS and MBAM will detect the 0.1 - 1.3 % missed by A2 and Avira?

In some way, this makes sense as MBAM did find 1.4%, so if it is the case that it is designed to catch what the others miss then okay. - How do they know what the others miss? What if a user has one of the AMs that got 90% or lower? SAS and MBAM cant possibly help then as they get between 1.4 - 5.6%

It would be very easy to test SAS and MBAM on the samples missed by the top five AM - just to see if its true that they detect what the others dont.......