Interesting Emsisoft Tests

[MikeNash]

Hi Guys,

Interesting test at Emsisoft with recent malware...

http://www.emsisoft.com/en/software/scanner/

Mike

[softtouch]

-{ Quote: "Hi Guys,

Interesting test at Emsisoft with recent malware...

http://www.emsisoft.com/en/software/scanner/

Mike" }-

It would be interesting when it would not have been on their own website, tested by them self. Of course, when I test my own products, and place the test result on my own website, my products would be the winner...

As I stated in another post, it detected 42 as malware on my pc, even the 42 have been all FP... so its detection rate was 100%, and other AV 0%...

[andyman35]

Interesting test results there,just a couple of points though.
If this test was against A2 own samples how come it didn't detect 100%?
Also testing MBAM and SAS against static samples doesn't give a true indication of their efficasy if that was how the test was performed,they're designed to detect running malware.

[MikeNash]

-{ Quote: "It would be interesting when it would not have been on their own website, tested by them self. Of course, when I test my own products, and place the test result on my own website, my products would be the winner...

As I stated in another post, it detected 42 as malware on my pc, even the 42 have been all FP... so its detection rate was 100%, and other AV 0%..." }-

Sure. If it's based on their own malware collection - you expect them to get 100%. But the _rest_ of the results, aside from the top spot are still interesting :)

[progress]

http://www.emsisoft.de/de/software/scanner/ ;)

[MikeNash]

-{ Quote: "Interesting test results there,just a couple of points though.
If this test was against A2 own samples how come it didn't detect 100%?
Also testing MBAM and SAS against static samples doesn't give a true indication of their efficasy if that was how the test was performed,they're designed to detect running malware." }-

At the bottom...

Note

This test was an Emsi Software internal comparison and does not represent an objective assessment of the detection performance of a-squared Anti-Malware. It stands to reason that our own in-house product will produce the best detection rate against our own Malware samples. After an in-depth analysis some files were rated as harmless in the meantime. That's why a-squared Anti-Malware did not reach 100%.

[CubonesCastle]

A-2 really should not have listed.

Since they did add all the tested samples to there databace before testing..... its kind of like me sending 40.000 malware samples to ClamAV waiting 4 months then testing. OFC CLamAV would have 99.99% detection rate.

[andyman35]

-{ Quote: "At the bottom...

Note

This test was an Emsi Software internal comparison and does not represent an objective assessment of the detection performance of a-squared Anti-Malware. It stands to reason that our own in-house product will produce the best detection rate against our own Malware samples. After an in-depth analysis some files were rated as harmless in the meantime. That's why a-squared Anti-Malware did not reach 100%." }-
OK thanks for answering point 1,but why test MBAM and SAS in this way that they're not designed to be used?

[MikeNash]

-{ Quote: "A-2 really should not have listed.

Since they did add all the tested samples to there databace before testing..... its kind of like me sending 40.000 malware samples to ClamAV waiting 4 months then testing. OFC CLamAV would have 99.99% detection rate." }-


I agree completely. The first place must be discounted - even in their own notes that say that "of course" they will come first. What's interesting for me is the rest of the pack.

[MikeNash]

-{ Quote: "OK thanks for answering point 1,but why test MBAM and SAS in this way?" }-

??? Dunno. I guess if it's a test of scanners, you'd run a scanner. That's one for Emsisoft, not for me.

[andyman35]

-{ Quote: "??? Dunno. I guess if it's a test of scanners, you'd run a scanner. That's one for Emsisoft, not for me." }-
Yes but a test should show a true reflection of the capabilities of the products tested in order to be credible.I put it that had these malware samples been activated on the system in question,both would have achieved a far higher percentage.Yes they're both scanners,but they're designed to scan for running malware.

[Franklin]

Four recent samples.

it021.exe - a-squared 4.0.0.101 2009.04.30 <- Not Detected - Win32.Malware.gen

6007_1.exe - a-squared 4.0.0.101 2009.04.30 <- Not Detected - a variant of Win32/Tinxy.AD

bb.jpg - a-squared 4.0.0.101 2009.04.29 <- Not Detected - Generic Dropper.cx

setupxv.exe - a-squared 4.0.0.101 2009.04.22 <- Not Detected - virus:FraudTool.Win32.MalwareRomovalBot.b

[MikeNash]

-{ Quote: "Four recent samples.

it021.exe - a-squared 4.0.0.101 2009.04.30 <- Not Detected - Win32.Malware.gen

6007_1.exe - a-squared 4.0.0.101 2009.04.30 <- Not Detected - a variant of Win32/Tinxy.AD

bb.jpg - a-squared 4.0.0.101 2009.04.29 <- Not Detected - Generic Dropper.cx

setupxv.exe - a-squared 4.0.0.101 2009.04.22 <- Not Detected - virus:FraudTool.Win32.MalwareRomovalBot.b" }-

Did the other vendors catch these?

[andyman35]

If I was of a suspicious persuasion I might think that showing MBAM and SAS (2 directly competing products) in such a poor light wouldn't be bad for business at all.::)

[Saraceno]

If a-squared came third, or last, doesn't bother me, it's only one test, so hopefully, no one will take it personally.

And the note explains it quite well - "This test was an Emsi Software internal comparison and does not represent an objective assessment of the detection performance of a-squared Anti-Malware. It stands to reason that our own in-house product will produce the best detection rate against our own Malware samples. After an in-depth analysis some files were rated as harmless in the meantime. That's why a-squared Anti-Malware did not reach 100%."

But just commenting on the one test - being in a two-week period, some of those AVs which aren't in the top group, may add malware signatures a few weeks later, as they might not have the staff and/or resources.

Launching malware would be an interesting test. I still think Emsisoft would do well in launching/preventing malware, many which might be missed by an on-demand scan, as their behaviour blocking (Mamutu/a-squared anti-malware) is a core component of their software.

But on-demand scanning, not bad - for a small company.

[Franklin]

-{ Quote: "Did the other vendors catch these?" }-
File it021.exe - Result: 4/40 (10.00%)

File 6007_1.exe - Result: 11/40 (27.50%)

File bb.jpg - Result: 10/40 (25.00%)

File setupxv.exe - Result: 2/40 (5.00%)

[MikeNash]

-{ Quote: "If I was of a suspicious persuasion I might think that showing MBAM and SAS (2 directly competing products) in such a poor light wouldn't be bad for business at all.::)" }-

Typically, that's why companies publish results that show their product in a good light, and forget (or attack) those that don't.

I don't think anyone here would be surprised at Emsisoft's motives for publishing these results.

[MikeNash]

-{ Quote: "
Launching malware would be an interesting test. " }-

That would be a very interesting test indeed. It would be hard to do it with a high number of samples - but it would be interesting to pluck out some of the ones that few companies detected and run them through by hand to see if it made any meaningful difference to the results.

[MikeNash]

-{ Quote: "File it021.exe - Result: 4/40 (10.00%)

File 6007_1.exe - Result: 11/40 (27.50%)

File bb.jpg - Result: 10/40 (25.00%)

File setupxv.exe - Result: 2/40 (5.00%)" }-

Nasty.

[andyman35]

-{ Quote: "Typically, that's why companies publish results that show their product in a good light, and forget (or attack) those that don't.

I don't think anyone here would be surprised at Emsisoft's motives for publishing these results." }-
This is why the independence of the testers is of paramount importance when ascertaining the credibility of any test.

[Saraceno]

I installed some legitimate software the other night, but when I noticed it was trying to change browser settings, at least gave me an idea how a 'clean' file is behaving.

So sometimes it's not only malware you want to prevent, you want to prevent unwanted system changes. Well to me, that's important.

Anyway, although this is an on-demand scan, many may agree, Mamutu/emsisoft's behaviour blocker, is easy to use and provides strong protection.

[andyman35]

-{ Quote: "That would be a very interesting test indeed. It would be hard to do it with a high number of samples - but it would be interesting to pluck out some of the ones that few companies detected and run them through by hand to see if it made any meaningful difference to the results." }-
Agreed 100%:thumb:

[MikeNash]

-{ Quote: "This is why the independence of the testers is of paramount importance when ascertaining the credibility of any test." }-

*grin* I couldn't agree with you more.

[Saraceno]

And same applies to Comodo, that's an extremely good result considering its a new AV. And launching the malware missed, the rest would most likely, be stopped in its tracks.

[emsisoft]

When preparing this 'test' we tried to be as careful as possible to avoid arguments like "this test is fake" and "the test methods are questionable".

It was not out intention to give an overall rating on security products. Of course we can't do that independently as a competing company.

We just wanted to show the differences of the on demand scanners and the speed of updates. Several independent tests recently showed that the top 5 scanner detections differ only by 1%, but these tests didn't show if they missed the same 1% or different 1% of the samples.

Another thing we wanted to uncover is the fact that antispyware tools, even if they come with "+AV" modules, can't really compete with the top antivirus players on the market.

Regarding Malwarebytes: This freeware scanner tool (not speaking about the guard version) is highly recommended by many people (not only here), but I have not seen a single test of their scanner capabilities so far. Some independent testers I've spoken with, told me that it is not good enough to be included in their tests at all, but that's another story. Well, we were very surprised that it wasn't able to detect at least 50% of our samples. I guess most users like MBAM because of the scan speed, but it's new to me that it is made to detect only running malware. Most people use only the freeware scanner without guard imho.

Again, our test is not a 'security' test that shows how good a program can actually protect your computer. It is a pure scan engine test. Not more, not less.

Testing was done very carefully and reproducable. If any of the vendors needs a hash list of the missed samples, please send me a note.

[andyman35]

-{ Quote: "When preparing this 'test' we tried to be as careful as possible to avoid arguments like "this test is fake" and "the test methods are questionable".

It was not out intention to give an overall rating on security products. Of course we can't do that independently as a competing company.

We just wanted to show the differences of the on demand scanners and the speed of updates. Several independent tests recently showed that the top 5 scanner detections differ only by 1%, but these tests didn't show if they missed the same 1% or different 1% of the samples.

Another thing we wanted to uncover is the fact that antispyware tools, even if they come with "+AV" modules, can't really compete with the top antivirus players on the market.

Regarding Malwarebytes: This freeware scanner tool (not speaking about the guard version) is highly recommended by many people (not only here), but I have not seen a single test of their scanner capabilities so far. Some independent testers I've spoken with, told me that it is not good enough to be included in their tests at all, but that's another story. Well, we were very surprised that it wasn't able to detect at least 50% of our samples. I guess most users like MBAM because of the scan speed, but it's new to me that it is made to detect only running malware. Most people use only the freeware scanner without guard imho.

Again, our test is not a 'security' test that shows how good a program can actually protect your computer. It is a pure scan engine test. Not more, not less.

Testing was done very carefully and reproducable. If any of the vendors needs a hash list of the missed samples, please send me a note." }-
It's common knowledge that MBAM isn't meant to be used as a replacement for a dedicated AV/AS,it's strength is detecting malware that has bypassed this layer and is already installed/running on a system,not detecting inert samples.

[emsisoft]

-{ Quote: "It's common knowledge that MBAM isn't meant to be used as a replacement for a dedicated AV/AS,it's strength is detecting malware that has bypassed this layer and is already installed/running on a system,not detecting inert samples." }-

Is that Wilders common knowledge? Can't find any word on the MBAM homepage that states that it is a complementary tool. If it is one, isn't it highly dangerous to not telling people that? People who think that it can replace their AV for $25 for lifetime..

[progress]

-{ Quote: "
Regarding Malwarebytes: This freeware scanner tool (not speaking about the guard version) is highly recommended by many people (not only here), but I have not seen a single test of their scanner capabilities so far. Some independent testers I've spoken with, told me that it is not good enough to be included in their tests at all, but that's another story." }-

I also think it's overrated, that's just my opinion :-X I remember this test (http://ssupdater.com/modules/Forums/index.php?showtopic=3746), the results are very similar to the emsisoft test! ::)

[andyman35]

-{ Quote: "Is that Wilders common knowledge? Can't find any word on the MBAM homepage that states that it is a complementary tool. If it is one, isn't it highly dangerous to not telling people that? People who think that it can replace their AV for $25 for lifetime.." }-
http://www.malwarebytes.org/forums/lofiversion/index.php/t8068.html
Please read the post by Nosirrah there.

[andyman35]

-{ Quote: "I also think it's overrated, that's just my opinion :-X I remember this test (http://ssupdater.com/modules/Forums/index.php?showtopic=3746), the results are very similar to the emsisoft test! ::)" }-
Strange how this overrated product is used by so many within the industry,including Symantec and Cyberdefender tech support.::)

[Franklin]

I'll just add that working on non-Wilders pc setups MBAM has helped me more than any regular av in wresting back control from some rogue infection and in fact getting the resident av enabled again.

Real world = real deal. ;)

When it comes to rogues MBAM kicks butt! :thumb:

Quote MBAM author:
-{ Quote: "MBAM is designed to work along side your existing antivirus software , not conflict or compete with it ." }-

[Bubba]

-{ Quote: "It was not out intention to give an overall rating on security products" }-But that is where the rub comes in, not only with this test but with other like tests where an "overall rating" was given to the public to chew on with very little testing methology given.

1) Was each program tested with what I'll call default settings ?
2) What was the actual numbers when it comes to files tested...."Trojans/Backdoors, Worms and Bots" <snip> "other types of Malware such as Viruses, Spyware, Adware, Rootkits, Keyloggers, Dialers, etc."

Just one small example....anti-malware program AAA does not by default installation detect remote access tools, password-cracking applications, and keyloggers. The test then would paint program AAA as something less than adequate in the eyes of users if default\out of the box was used.

Bottom line, tests such as this with inadequate testing explanation is subject to the negative criticism it generates and does not serve the security community. It's simply a sales pitch.

Bubba

[emsisoft]

-{ Quote: "http://www.malwarebytes.org/forums/lofiversion/index.php/t8068.html
Please read the post by Nosirrah there." }-

Thanks for the explanation. It was even new to me, as I didn't spend much time with MBAM so far.

-{ Quote: "
MBAM is antimalware software that is designed to fill in the gaps left by antivirus software.
" }-

Why the .... isn't that stated in big capitalized letters on their main page? It's confusing users. But.. from a marketing point of view it's a great strategy of them, not telling the most important fact!

[Saraceno]

Malwarebytes, say even if it has two per cent overall detection, it seems to get those problems files that are running when a user is using a leading AV.

Even if it's just one or two files - those few files are being downloaded by users everywhere (msn etc) but a lot of the time, are not picked up by the leading programs. So it has its place.

So I agree with andyman, it's an add on tool.

But Christian has pointed out a very valid point, MBAM might not be advertising/marketing their product as well as they could.

[Saraceno]

To me, smaller companies have to work harder to win over consumers. Somewhat of a sales pitch I agree, but I still maintain, emsisoft has a valuable product.

So I wouldn't say its false claims by any means, just promotion. It's what smaller companies do.

But there aren't too many tests floating around, so rather than react negatively, I just add this test to the rest of tests which help form some of my own opinion.

[emsisoft]

-{ Quote: "1) Was each program tested with what I'll call default settings ?
2) What was the actual numbers when it comes to files tested...."Trojans/Backdoors, Worms and Bots" <snip> "other types of Malware such as Viruses, Spyware, Adware, Rootkits, Keyloggers, Dialers, etc."
" }-

1) Yes we used default settings. That's the only way to represent real world usage.

2) Didn't count the malware names for category. We simply used all malware samples that we've got during the first 15 days of April, except those who were submitted by virustotal and jotti. There is no special focus on any specific category, it's real life malware. But if you want I can provide a scan log of a-squared that shows all names.

[andyman35]

-{ Quote: "Thanks for the explanation. It was even new to me, as I didn't spend much time with MBAM so far.



Why the .... isn't that stated in big capitalized letters on their main page? It's confusing users. But.. from a marketing point of view it's a great strategy of them, not telling the most important fact!" }-
That's a fair point,it should be made crystal clear the intended useage to avoid misunderstandings.

[Saraceno]

Without going off-topic, if a company feels they are doing well against new threats, whether that be Dr.Web, Prevx, Avira, MBAM, I don't see any harm in publishing results.

End of the day, many of the products we all use here don't have the same luxury of being pre-installed on users machines like others, on company and government systems, and earning the mega bucks.

[Az7]

Special thanks to T3.

[progress]

-{ Quote: "That's a fair point,it should be made crystal clear the intended useage to avoid misunderstandings." }-

So a-squared Anti-Malware is AV/AS/BB and Malwarebytes Anti-Malware is just a pure AS?

[Saraceno]

From my understanding, yes.

[progress]

So maybe it should be better called Malwarebytes Anti-Spyware or Malwarebytes Anti-Rogueware, I also thought that Malwarebytes Anti-Malware is a full package like A² Anti-Malware :-\

[Saraceno]

I think rogue software is definitely its specialty. And its thorough removal of rogue threats. I do think MBAM's website should show screenshots, list the types of rogue programs it finds, overall, should promote its product more than it currently does.

[LoneWolf]

-{ Quote: "

Bottom line, tests such as this with inadequate testing explanation is subject to the negative criticism it generates and does not serve the security community. It's simply a sales pitch.

Bubba" }-

Agree 100%

[emsisoft]

That's what it was intended to be - marketing.

The page doesn't claim that it is an objective or independent 'test'. But of course, it might be interesting for some of us here.

[andyman35]

-{ Quote: "So a-squared Anti-Malware is AV/AS/BB and Malwarebytes Anti-Malware is just a pure AS?" }-
I wouldn't classify it as a standard AS,the developers say it targets a wide range of malware (rogues etc) that bypass traditional security products.Therefore I suppose a targetted AM tool would be a fair description.It contains 81410 'fingerprints' used for detection,this relatively small database implies to me it picks up malware traces by a forensic type analysis,not limited to just standard Spyware traces.

Quote by Nosirrah.

"The reasons are many but the critical ones are the "rules" AV software is bound by . AV software is forced to detect malware by examining file contents alone and while this usually works there is a lot of malware that rotates their obfuscation tricks so often that the AVs simply cant keep up . AV software also often does not work to undo system damage left behind by malware , an area that MBAM is very good in . MBAM does look at file contents but that is only one of many ways we can detect a file and this is why we can hit a lot of malware that the AVs miss . Lets say we detect an infection by 6 points of contact . If 5 of those were to change we would still detect that infection completely . If any of those 5 were file contents AV software would fail to detect the changed malware . MBAM also has many family specific checks it does to heuristically detect common but poorly detected malware that AV software cant even come close to matching ."

[LoneWolf]

-{ Quote: "That's what it was intended to be - marketing." }-

Of course it was.

[Saraceno]

Just a company test. No more no less.

Come on fellas, ;) last I checked, marketing isn't an offence. Otherwise they'd shut down the net, tv and radio.

Yeah it has some false positives, but their programs are solid. Their free programs alone, hijackfree and a-squared free, are awesome tools. I could remove most malware with just the free tool hijackfree. Try it and you'll most likely agree.

Some people are paying big money for security, while this company is giving this stuff for free. I can't complain. Happy free and paid customer here. :thumb:

Lastly, the point of this test to me is, a-squared free version, which is free for anyone, can remove a lot of threats. Probably more than programs people are paying for.

[trjam]

also, wouldnt the settings for each product play into its ability.

[Chubb]

I just can't believe that Sophos got only 33.2% detection rate.

[Saraceno]

It's only on threats during a two week period.

Sophos might process new threats at a later stage, or slower than the others.

[progress]

-{ Quote: "
Some people are paying big money for security, while this company is giving this stuff for free. I can't complain.
" }-

:thumb:

[CubonesCastle]

Just a quick note incase this has not been noted before, MBAM only adds samples that other vendors don't detect. If it has more then 10 flags at VT its not gonna get put on the list.

I think they do this because it makes the program less heavy and works better along with a normal AV scanner.

[MAOS]

another useless test out there.

I'm still laughing reading the note:

-{ Quote: "
This test was an Emsi Software internal comparison and does not represent an objective assessment of the detection performance of a-squared Anti-Malware. It stands to reason that our own in-house product will produce the best detection rate against our own Malware samples. After an in-depth analysis some files were rated as harmless in the meantime. That's why a-squared Anti-Malware did not reach 100%.
" }-

After?!?! LOOL This test must be really serious ;D

And, by the way, are you going to fight malwares with Mamutu? ;D Totally user mode software that is supposed to trace software behaviors?

It could be bypassed in 0.1 seconds ;D

[aigle]

A useless test. Good advertisement for average users. A-sq has a lot lot false positives. Unimpresive.

[Warklen]

-{ Quote: "A useless test. Good advertisement for average users. A-sq has a lot lot false positives. Unimpresive." }-

Completely agree

[Martijn2]

As far as I see it this test is not a advertisement for A-squared, it's purpose is to see how fast anti-virus programs add new samples.

But as expected Avira on the top again!

[raven211]

-{ Quote: "A useless test. Good advertisement for average users. A-sq has a lot lot false positives. Unimpresive." }-

If this is true, which I believe it's according to many users' comments in different topics - :thumb:

[rdsu]

-{ Quote: "A useless test. Good advertisement for average users. A-sq has a lot lot false positives. Unimpresive." }-
Well said! :thumb:

And also a resource hog... ;)

[RubbeR DuckY]

-{ Quote: "
MBAM only adds samples that other vendors don't detect. If it has more then 10 flags at VT its not gonna get put on the list.
" }-

This is incorrect. We prioritize those that are less detected but add all samples.

These types of tests are just great! I love the smell of bs in the morning. If your going to believe our program and SUPERAntiSpyware caught below 10% of samples then you need to do some Google research. And yes, I am defending the competition.

What the testers did is as follows. Gather samples, place them into a folder, right click the folder and scan with the utility to see how many threats it finds. Now if the test was done on live infections, I assure you that graph would be reverse, i.e. a-squared on bottom and SUPERAntiSpyware on top.

Believe what you want. Google is the answer. There is a reason why SAS and MBAM are recommended multiple times -- because they work, not because they perform well in a lab environment tested by the company that comes in front ;).

[RubbeR DuckY]

Mike,

I think it is also fair to fully disclose that Emsisoft helps distribute (or sell) your product, Online Armor as seen here (http://www.emsisoft.com/en/software/free/). I assume that is not the reason you posted this thread, however, full disclosure is always appreciated!

[Phenom]

no way a-squared got 99.9%...

[LoneWolf]

-{ Quote: "
These types of tests are just great! I love the smell of bs in the morning." }-

Somebuddy had to say it.

[SUPERAntiSpy]

-{ Quote: "When preparing this 'test' we tried to be as careful as possible to avoid arguments like "this test is fake" and "the test methods are questionable".

It was not out intention to give an overall rating on security products. Of course we can't do that independently as a competing company.

We just wanted to show the differences of the on demand scanners and the speed of updates. Several independent tests recently showed that the top 5 scanner detections differ only by 1%, but these tests didn't show if they missed the same 1% or different 1% of the samples.

Another thing we wanted to uncover is the fact that antispyware tools, even if they come with "+AV" modules, can't really compete with the top antivirus players on the market.

Regarding Malwarebytes: This freeware scanner tool (not speaking about the guard version) is highly recommended by many people (not only here), but I have not seen a single test of their scanner capabilities so far. Some independent testers I've spoken with, told me that it is not good enough to be included in their tests at all, but that's another story. Well, we were very surprised that it wasn't able to detect at least 50% of our samples. I guess most users like MBAM because of the scan speed, but it's new to me that it is made to detect only running malware. Most people use only the freeware scanner without guard imho.

Again, our test is not a 'security' test that shows how good a program can actually protect your computer. It is a pure scan engine test. Not more, not less.

Testing was done very carefully and reproducable. If any of the vendors needs a hash list of the missed samples, please send me a note." }-

Your test is simply a one sided test to make your product look good - bottom line. Provide the COMPLETE sample set so we can run our own test against the sample set.

Here is a except from my blog regarding testing methodology that readers wish to consider when reviewing the test above.

The Importance of Testing Methodology

In today’s oversaturated market of anti-spyware/malware/adware applications, it is becoming increasingly difficult for users to determine which applications will perform best for their specific needs. Thus, they look for standardized and legitimate “comparative tests” of these applications.

Testing anti-spyware applications is not an easy task. It is imperative that those who are going to undertake the task of testing need to have the skills to perform the tests competently and to test the products in real-world situations. Otherwise they are not performing a service to users. Users also need to examine the credibility of the party testing the applications and not simply “look at the numbers.” Currently, most tests are not comparing “apples to apples” because every anti-spyware application uses different methods of reporting the “numbers” of infections detected and removed.

There are standardized and widely accepted elements of any investigative report. These include an Introduction, Materials and Methods or Procedures, Results, Discussion and usually, but not always, Conclusions.

The most critical elements of an adequate report or investigation are to provide the reader with the Materials and Methods used which would allow others to duplicate the experiment or investigation to determine the validity of the results; that is, are the results reproducible in the hands of others using the same procedures (Materials and Methods). Thus, the methodology used in any investigation must be of sufficient detail to allow any interested parties the opportunity to independently validate the results.

In using non-standardized methods, it is critical to provide detailed procedure in order to ensure validation by allowing reproduction of the results by others.

When or if it is determined that the methodology is itself flawed or contains documented errors, this invalidates the results and casts serious concerns on other components or elements of the entire methodology and on the results.

The level of detail cannot be assumed or taken on faith. Therefore, it is of the utmost importance to provide a level of detail which removes any ambiguity as to how something was done and to provide and detail the safeguards used to ensure that the procedures were indeed followed.

In examining the testing methodology used by recent tests by A-Sqared, it is unclear whether their own procedures were followed when flaws or errors are discovered as detailed elsewhere. This casts doubt on how other elements were carried out. Furthermore, it is one thing to say how you are carrying out the testing and another to actually follow the protocol. Thus, alleged transparency by providing the purported methodology cannot in and of itself be accepted on faith and can be extremely misleading particularly in view of any demonstrated inadequacies.

Malware testing is certainly a daunting task and adequate documentation of methodology is the single most important element in validation of the results. When testing is performed by individuals one can accept and or excuse minor inadequacies. However, when the results are performed by alleged experts, in testing facilities which exist for testing purposes, they must be held to the highest standards.

[trjam]

I think it is CK that Emsisoft has products that are FP magnets. It is no wonder they detect everything. Now, if they ever correct this, they will either be almost as good as Eset, or they will have no detection at all. So, Nick, we all know SAS is good.

Bottom line: Emsisoft should have kept this test internal. It will cost them.:wacko:

[trjam]

-{ Quote: "I don't know, that's a long essay haha. But yeah, how do you explain Avira always detecting over 98% in all the tests? Can't just be coincidence right?

By the way, thanks for releasing a Superantispyware update! Just updated my signature." }-
Oh, give me a break, Avira is the 2nd FP Queen in the software world.

[fcukdat]

-{ Quote: "Mike,

I think it is also fair to fully disclose that Emsisoft helps distribute (or sell) your product, Online Armor as seen here (http://www.emsisoft.com/en/software/free/). I assume that is not the reason you posted this thread, however, full disclosure is always appreciated!" }-

Your too polite Marcin :)

If it walks like a duck and quacks like a duck then it most probaly is ::)

Christian,

There is a very simple reason why both MBAM and SAS are widely used by malware removal experts at help forums as the principal tools and not your a2 free.

Realworld infections and software performance against them do not reflect your contrived test results.

Are they really eating away at your target market so much that you have to result to this kind of blatent misrepresentation inorder to promote your software ?

Mike,

As an expert in your field i am quite simply surprised that you can even think those test results carry any credebility:thumbd:

[BrendanK.]

*sigh* I thought Emsisoft would be different...:-\

I agree that this test should of been kept "in house", but if Emsisoft cared about it's users I think they would still rectify themselves and remove the test from the website. There have been independent tests which show A2 top of the charts or equal too and almost greater then number 1, which could be used on their website to show that A2 is a very good antivirus.

I hope you guys haven't gone over your head just because you got an addition from Ikarus.

[ypestis]

-{ Quote: "I also think it's overrated, that's just my opinion :-X I remember this test (http://ssupdater.com/modules/Forums/index.php?showtopic=3746), the results are very similar to the emsisoft test! ::)" }-


Very similar, and can be repeated by downloading their latest "do it yourself
malware test".
The real point is the outcome of both test are similar because the same
methodology is used.
Static,un-executed malware, is scanned by a application that is primarily designed to detect running malware.

[trjam]

-{ Quote: "Mates, just relax. Mike was merely posting some "interesting" tests. Nothing more, nothing less. We can interpret it at our own leisure." }-

That is pure tee bull ****. Sorry for my venting. But you are going to come here and say "Mike was posting some interesting tests, nothing more nothing less."

How about folks actually believe this stuff, and as a vendor, dont play dumb. How about the fact that some feed their families or dont, because of tests, err, excuse me, "interesting" tests.

All I can say is, you want to have office parties, then do it, but in the office. Any test that can not be validated is not a test, but a scam. You folks really screwed up today. I am personally asking others to boycott this vendor. You dont play around with others lives.:thumbd:

[RubbeR DuckY]

It is a shame that a reputable company such as Emsisoft would be so desperate to suck money out of their customers that they would resort to discrediting other reputable companies with a test clearly suited to put a-squared in first place. Just business I guess right? What's next, a toolbar? I heard that's a great business move too.

Personally, I'm touched. As the owner of a company that I know will outperform a-squared anti-malware in a live test I am just sitting here with a huge grin knowing they had to resort to this.

Customer happiness > Money in our opinion. Maybe not theirs.

[RubbeR DuckY]

-{ Quote: "At the end of the day, it's up to the user to do their own personal un-biased research and attempt to interpret whatever is being advertised." }-

Agreed! Oh the power of a search engine is so great :).

[Dr33]

snake oil :argh:

[rdsu]

-{ Quote: "I don't know, that's a long essay haha. But yeah, how do you explain Avira always detecting over 98% in all the tests? Can't just be coincidence right?" }-
Please, try to understand better how program works, and then you will have the answer for this... ;)

[rdsu]

-{ Quote: "Please enlighten me if you have time and can be bothered. Thanks." }-
;D You can start by searching in SUPERAntiSpyware Forum... ;)

About this topic:
When programs are just great and do their jobs, like SUPERAntiSpyware and MalwareByte's Anti-Malware, they doesn't need to show up here and there with this kind of BS tests...

It's really incredible how some "respectable" security companies can do things like this...

[trjam]

-{ Quote: "Hmm, yes I see your point mate, and can even understand your frustration to an extent.

However, it seems a bit extreme to be using terms like "others lives". This is security software. There are always advertisements for it. Just like on TV, there are always advertisements for this and that. They make it sound good, they make it look good.

At the end of the day, it's up to the user to do their own personal un-biased research and attempt to interpret whatever is being advertised." }-
I agree, but if a software is failing due to published tests showing it is crap, then heads do roll. So all tests need to be very accurate or yes, you could be messing with someones future.

[MikeNash]

-{ Quote: "Mike,

I think it is also fair to fully disclose that Emsisoft helps distribute (or sell) your product, Online Armor as seen here (http://www.emsisoft.com/en/software/free/). I assume that is not the reason you posted this thread, however, full disclosure is always appreciated!" }-

Yes - we're working with Emsisoft - I've previously announced this here and even issued press releases to that effect. It's not a secret.

There has been a lot of discussion here about the new Comodo AV. This test contains Comodo (which does surprisingly well) and I thought that folks here would be interested in the results.

It seems I managed to avoid generating yet another OA vs CIS thread, but instead created an everyone versus everything thread instead...

[rdsu]

-{ Quote: "There has been a lot of discussion here about the new Comodo AV. This test contains Comodo (which does surprisingly well) and I thought that folks here would be interested in the results.

It seems I managed to avoid generating yet another OA vs CIS thread, but instead created an everyone versus everything thread instead..." }-
Did I read well!? ::)

[RubbeR DuckY]

Mike, many are unfamiliar with those press releases, including myself. Hence, full disclosure is kindly appreciated :).

[CubonesCastle]

-{ Quote: "It is a shame that a reputable company such as Emsisoft would be so desperate to suck money out of their customers that they would resort to discrediting other reputable companies with a test clearly suited to put a-squared in first place. Just business I guess right? What's next, a toolbar? I heard that's a great business move too.

Personally, I'm touched. As the owner of a company that I know will outperform a-squared anti-malware in a live test I am just sitting here with a huge grin knowing they had to resort to this.

Customer happiness > Money in our opinion. Maybe not theirs." }-


Honestly, I purchased both Malwarebytes and Superantispyware a while ago just because you both have good atitudes have good support and generally work quitely in the background like a good company should.

Also RubbeR DuckY while your here make an auto-updater for malwarebytes it would help.

[SUPERAntiSpy]

-{ Quote: "Yes - we're working with Emsisoft - I've previously announced this here and even issued press releases to that effect. It's not a secret.
" }-

FYI, A Google News Search revealed nothing unfortunately...
http://news.google.com/news?hl=en&q=emsisoft%20tall%20emu&um=1&ie=UTF-8&sa=N&tab=wn

[andyman35]

It's actually not that difficult to test how effective SAS and MBAM are in a realistic way.Simply by loading up a VM with every rogue and drive-by,etc that you can find then running these tools.The results are not a tiny percentage of the malware discovered but,in fact the overwhelming majority of the threats neutralised.

http://remove-malware.com/antimalware/anti-malware-reviews/malwarebytes-pro-removal-and-detection-review/

http://remove-malware.com/antimalware/anti-malware-reviews/superantispyware-detection-and-removal-video-nov-test-2/

[Tarnak]

-{ Quote: "FYI, A Google News Search revealed nothing unfortunately...
http://news.google.com/news?hl=en&q=emsisoft%20tall%20emu&um=1&ie=UTF-8&sa=N&tab=wn" }-

Hi, added one word to your search paramaters, voila

http://www.google.com/search?um=1&ned=us&hl=en&q=emsisoft+tall+emu+%2B+Wilders&btnmeta%3Dsearch%3Dsearch=Search+the+Web


Emsi and Tall Emu announce tech alliance > http://www.wilderssecurity.com/showthread.php?t=234937 ;D

[SUPERAntiSpy]

-{ Quote: "Hi, added one word to your search paramaters, voila

http://www.google.com/search?um=1&ned=us&hl=en&q=emsisoft+tall+emu+%2B+Wilders&btnmeta%3Dsearch%3Dsearch=Search+the+Web


Emsi and Tall Emu announce tech alliance > http://www.wilderssecurity.com/showthread.php?t=234937 ;D" }-

Oh, forum announcement, not press announcement. Lots of people read those! LOL :)

[MikeNash]

-{ Quote: "Oh, forum announcement, not press announcement. Lots of people read those! LOL :)" }-

http://lmgtfy.com/?q=emsi+tall+emu+press

[SUPERAntiSpy]

-{ Quote: "http://lmgtfy.com/?q=emsi+tall+emu+press" }-

Ok, so what is going to be done in removing SUPERAntiSpyware and MBAM from those results - that test is an absolute fake - essentially defaming SUPERAntiSpyware and MBAM - you know full well our products work very well - remember before A2 you and I had conversations about licensing.

[CubonesCastle]

I want to see a SAS + MBAM aliance, Develop a security suite and i'm sure it would be a gigantic contender both engine + A new AV engine would be something that most people would buy because more people then ever are more interested (Antispy>Antivirus) Because of cyber crime.

You guys already seem friendly. Just throwing it out there :D

[SUPERAntiSpy]

-{ Quote: "I want to see a SAS + MBAM aliance, Develop a security suite and i'm sure it would be a gigantic contender both engine + A new AV engine would be something that most people would buy because more people then ever are more interested (Antispy>Antivirus) Because of cyber crime.

You guys already seem friendly. Just throwing it out there :D" }-

It's a great idea because both SAS and MBAM have different strengths.

[CubonesCastle]

-{ Quote: "It's a great idea because both SAS and MBAM have different strengths." }-

Indeed i want to see it, I can tell you now. I would buy it.

[andyman35]

-{ Quote: "Indeed i want to see it, I can tell you now. I would buy it." }-
Looks like you might have started something very interesting there.;)

[MikeNash]

-{ Quote: "Ok, so what is going to be done in removing SUPERAntiSpyware and MBAM from those results - that test is an absolute fake - essentially defaming SUPERAntiSpyware and MBAM - you know full well our products work very well - remember before A2 you and I had conversations about licensing." }-

Hi Nick,

You seem to be under the mistaken impression that I am responsible for these results. The person you need to talk to here is Christian Mairoll at Emsisoft.

If you believe the tests are fake - take it up with them. I've explained why I posted the test, and I'm not going to get caught up in an inter-vendor pissing contest.

As for the licening question - yes, we did have conversations about it. As I recall, we reached agreement in principle in fact and it looked promising - but then I gave up on the idea and decided to scrap the idea of adding your product and stick with Kaspersky.

To avoid any misunderstanding - the reasons why we stopped our discussions with Nick were not in any way product related. I don't have any axe to grind with Nick or the MBAM guys (in fact this sad little thread is my first interaction with MBAM)


Mike

[Tarnak]

-{ Quote: "Looks like you might have started something very interesting there.;)" }-

Synergy > http://en.wikipedia.org/wiki/Synergy :)

[andyman35]

-{ Quote: "Synergy > http://en.wikipedia.org/wiki/Synergy :)" }-
Very apt description.;)

[Phenom]

LOL a-squared.. if emsisoft was to be in a av comparatives test they wouldn't even get award because of all the false positives. With ikarus the detection rate went higher, but remember when emsisoft didn't have the ikarus engine the detection rate was low, slow, memory hog.

[arran]

-{ Quote: "snake oil :argh:" }-


just like sandboxie in your sig which is also snake oil

[Franklin]

-{ Quote: "just like sandboxie in your sig which is also snake oil" }-
Thanks for that great bit of advice arran.

After several years of using Sandboxie and testing a few gig of malware samples with only a minor quirk here and there I will get rid of it completely.

Like eff I will, Sandboxie is one of if not the best security app ever created! :thumb:

[BrendanK.]

I'm sorry but this thread has turned into a brawl with everyone throwing punches at each other. Let's try stick to the topic here as we all want this issue sorted out, and we don't want the thread to be closed because as far as I can tell everyone wants to discuss this further :)

Now I may be young, but I have learned this in life so far...

1)Making friends and becoming stronger together will break down any foolish thing that comes in your way.
2)As well as rectifying your mistakes, everyone deserves a second chance. But if that chance is not used that is the bad luck of the person(or company in this case).
3)People have a right to express their own opinions without everyone else bringing hell down on them.

So please let's all take a chill :)

[Franklin]

Nothing like a bit of a punch-up every now and then to clear the air. ;)

And if you may discern that I get a tad shirty when anyone craps on Sandboxie then all I can say is "too bloody right I do". ;D

[progress]

No sandboxie rubbish please, you can talk about sandboxie here (http://www.wilderssecurity.com/showthread.php?t=240206)! Back to topic ;)

-{ Quote: "I think it is CK that Emsisoft has products that are FP magnets. " }-

Edit: What about MBAM and FP (http://www.malwarebytes.org/forums/index.php?showforum=42)?

Don't get me wrong, I use both tools (A2 / MBAM) ;)

[emsisoft]

Funny to see all the bashing early in the morning, wilders will never change.. ;)

Why don't you simply read the WHOLE text on the test page?

- It's clearly stated that this test doesn't say anything about a-squared's scanning capabilities because we used our own samples. In this view, Avira is the clear 'winner' and MBAM is the clear loser because they were not able to manage signature creation for more than 39,000 malwares in the wild. As said before, even I didn't know that MBAM are meant to detect ONLY active malware. Their website doesn't tell anything about it - it's sold like all other antivirus tools and as a first time visitor I'd believe that too. MBAM: Tell your customers the truth about your software abilities and we would start recommending it as a great addition to a-squared!

- It's clearly stated that this test is just a scanner and update speed test, not an overall security test. So why bash on it? Check out the results and make your own opinion about the tested things (and not about other things that are not part of this test!). Maybe we'll test the execution/blocking capabilities in the future.

- It's clearly stated that we used malware samples that we have got in our analysis dpt during April 1 and April 15. That means ALL samples. Not a special set that prefers a specific vendor. Just all, except those files that are spread by virustotal and jotti because it would simply blow up the test test unnecessarily. It is absolutely normal that signatures change within 3 weeks. Some files are shared malware files, some are FPs and some can not be considered as malicious enough to keep them in detection - that's why a-squared didn't reach the 100%. But keep in mind: Files that we don't detect any longer are still detected by others which helps them to get higher detection rate. I'd love to see the same type of test of all other vendors!

To make the test methodology more transparent, I can provide file hash lists to all tested vendors. Just send me a PM.

Regarding a-squared false positives: In early versions, we had a lot - true. No wonder - the Ikarus engine was made for the Austrian market only. They simply didn't get international feedback about FPs. That has changed since we partner with Ikarus. The latest independent test of av-test.org (published in the German ComputerBild magazine) showed that a-squared has less FPs than Kaspersky.

Guys, do me a favor and stop repeating old impressions over and over. Before bashing, give it a chance and re-run our current scanner and compare with the others. FPs are a major problem for all antivirus vendors in the meantime, not only for us. For sure there are FPs, daily, but they're fixed quickly.

[arran]

I'm sorry for bringing up sandboxie,in this thread. I just had it on my mind at the time.

Back on topic.

Regarding how legitimate the test results really are I chalenge the venders posting in this thread to have their products tested at av comparatives then we will all know for sure.

[BrendanK.]

-{ Quote: "I'm sorry for bringing up sandboxie,in this thread. I just had it on my mind at the time.

Back on topic.

Regarding how legitimate the test results really are I chalenge the venders posting in this thread to have their products tested at av comparatives then we will all know for sure." }-

They have already been tested by independent groups ??? And they did very well ranking in the top spot if not in the top 3. As for MBAM and SAS, I'm not sure of any tests. But I do know they protect me like nothing else can (of course they are a part of my layered approach and are not my only app which I love and could not live without, but for the spot they hold in my layers of security they are the best) :D They detect the stuff others miss, so for them to detect every possible malware sample is just plain stupid. I like MBAM and SAS how they are regardless of any test.

-{ Quote: "Malwarebytes' Anti-Malware can detect and remove malware that even the most well known anti-virus and anti-malware applications fail to detect." }-

My faith is loyal :P

[blacknight]

Personally I'm always surprised to read all this FP from A-Squared. I've been used it free for regular on demand scans from many years and I had very few FP. I use A-Squared as control scan of my system, and it's very powerful and reliable.

[BrendanK.]

-{ Quote: "Personally I'm always surprised to read all this FP from A-Squared. I've been used it free for regular on demand scans from many years and I had very few FP. I use A-Squared as control scan of my system, and it's very powerful and reliable." }-

Agreed. I have had a few false positives from time to time, but the same can be said for the other programs I have used.

[arran]

Actually that test said it is "NEW" Samples so maybe Super Antispyware isn't good with "NEW" Malware samples but it has a good detection rate for all other malware??

[Saraceno]

I don't think some people read the whole page - just looked at the graph and then saw red.

From my understanding, Emsisoft already had the samples so it's a given they would have added these.

Programs like Avira, Kaspersky, G-Data, and Avast to me were the clear winners. They were able to detect files that Emsisoft already had and knew were malware.

1. Avira
2. G-Data
3. Kaspersky
4. Avast
5. Norton
? Emsisoft (a-squared)

And to say right-clicking on a folder is a poor test, I think you have to realise it for what it is, just an on-demand test. People here are always right-clicking and scanning files. I right-click and scan files recovered from sandboxie. I right-click and scan through a bunch of files I've copied on to USB. I don't launch these programs, I'm right-clicking and scanning the files.

So right-click and scan, is still an important test, not the most important, but still, it's important.

If the test didn't state it was a simple right-click and scan, then I can see people getting upset.

I'm a big fan of almost all the products in the test, including SAS, MBAM and Emsisoft. But I'm not going to swear my loyalty to a product by carving the company name on my chest and going bananas when someone says otherwise.

Just relax, if the CEO Christian isn't aware of MBAM or SAS and their capabilities, then he's missing out on observing some valuable security programs.

If you bash on a company, then you're only displaying the same behaviour as you're alleging they displayed.

[MAOS]

Whatever, do you really believe to a company who release a comparative test and they do not have even tested samples they are using for testing? And they are so smart to add a note saying that "after in depth analysis some samples are not considered malicious"?

And I will not comment about their superb Mamutu... ;D

[Saraceno]

You've proven my point about bashing a product, 'superb Mamutu'.

Please release your thorough tests of Mamutu, which a lot of people here use, can't wait to read them. ;)

Also, I'm no AV expert, but I remember reading how long it takes Dr Web researchers to analyse some files. So I'm gathering, it does take some time determining whether a file is completely clean.

[Kees1958]

-{ Quote: "
Guys, do me a favor and stop repeating old impressions over and over. Before bashing, give it a chance and re-run our current scanner and compare with the others. FPs are a major problem for all antivirus vendors in the meantime, not only for us. For sure there are FPs, daily, but they're fixed quickly." }-

Christian,

First congrats with the OA alliance. Secondly it is a perfect explanation that Ikarus scored high in FP's due to the lack of international operation. Consumer to consumer communication exactly describes your reputation, so you need to take other measurements, like publishing those low FP reviews on your website also.

With themed messaging (this test and new low FP's messages) you work on your company's and product's identity. It really pays off when you align your communication with the release calendar of your products. Simply asking people to change their minds won't work.

Publishing a test which is completely dominated by your own test set, is not an independant comparison, because the priciples on which it is performed, are benifial to A2's developers team. To other vendors it would trigger the same reaction as the stupid test of Mamuto on Matousec. I would not impose something on others, which you do not want to impose on yourself.


Regards Kees

[MAOS]

If analyzing malicious softwares takes a lot of time, then it doesn't allow you to do and release comparative testing and then fix them because some malicious softwares were not really malicious after analysis.

When you've done your research and you're sure about samples you are going to test then you can do comparative testing.

By doing so you're only showing your absent skills.

About Mamutu: what you're expecting from a totally user mode software which applies its hooks only to Win32 subsystem API? Yes, it can gather behaviors from executed software.

Yet, it can be easily bypassed and many malwares are able to analyze themselves and fix API hooks or totally bypass them by calling directly the dispatcher.

Everyone in the security industry knows exactly how bad can be using user mode API hooking.

[NoIos]

I will never get tired repeating that the combination a-squared + Ikarus is one of the best I have ever seen in many years. Using it on every day conditions as an on demand scanner...the last 5 months has missed only a few malware in downloaded files. Some FPs yes...but the rate is really small.

Now, I think it was an error that Mike Nash came here to inform us about the test. It's not bad to provide informations and personally thank him for his post, but he should act and think more professionally before posting. Most of us already knew about the Emsisoft + Tall Emu "alliance", so it didn't seem that elegant see him posting about it. Of course he's a free man...he can do whatever he likes.

MBAM and SAS have my respect too. They do a fine job. The only problems I had with MBAM was some false positives with fake drivers and SAS...although I had reported some files many times as false positives...they got flagged by their team as non malware months later.
For the rest they work great. I agree with all those who say that MBAM and SAS have a different point of view about security, so normally they don't fit in a regular AV test.

The problem with MBAM, SAS and a-squared is that they have not managed till now to convince me that I need their real time protection...having Avira or Avast installed + a firewall with hips. All 3 programs are great as on-demand scanners.

[Smokey]

-{ Quote: "
- It's clearly stated that this test is just a scanner and update speed test, not an overall security test." }-
And, most important, just an internal comparison. No more, no less.

-{ Quote: "Check out the results and make your own opinion about the tested things (and not about other things that are not part of this test!). Maybe we'll test the execution/blocking capabilities in the future." }-
How about removal capabilities? Apparently not part of your test, nevertheless important. Anyway, to me.

<S>

[Saraceno]

MAOS, remember, most people using computers/laptops aren't in the security industry. You're talking about 1 per cent or less of users.

A product that appeals to the 1 per cent might cripple a novice user. So different products are needed for different users.

[emsisoft]

I expected that some vendors who are not happy with the results would allege that our test is fake. To all of them: Don't try to terrorize us with lawyer threats, better tell us what else information we should present to make our testing methodology 100% transparent.

Here is the hash list of the tested samples:
http://www.emsisoft.com/en/software/scanner/antivirustest_a2samples_200904_hashes.txt

And here is the list of malware names that a-squared detected:
http://www.emsisoft.com/en/software/scanner/antivirustest_a2samples_200904_names.txt (that list contains a bit more lines than scanned files because some samples were packed and the scan log contained 2 or more lines per file)

We did not test every one of the 40,000 samples if they are still able to run. No tester can do that for that huge amount of malware. But remember: These samples were worth to be included in our signatures, so you can presume that they're not just harmless ascii files. The majority of all antivirus programs detected them too.

Interesting note: Even if you pick only the major spread ones (conficker, etc), they're much more samples than MBAM detected at all (~550 of 40k).

[MAOS]

-{ Quote: "MAOS, remember, most people using computers/laptops aren't in the security industry. You're talking about 1 per cent or less of users.

A product that appeals to the 1 per cent might cripple a novice user. So different products are needed for different users." }-

You asked me for an explanation, and I've given a technical one to you. I know that 99% of people will not understand that, this is why I've just written in simple words that Mamutu isn't a good software and it can be easily bypassed.

This one should be easier to understand

[MAOS]

-{ Quote: "
We did not test every one of the 40,000 samples if they are still able to run. No tester can do that for that huge amount of malware. But remember: These samples were worth to be included in our signatures, so you can presume that they're not just harmless ascii files. The majority of all antivirus programs detected them too." }-

Bad one, my friend. This not the right answer. If you are not able to confirm the nature of samples you are going to test you can't release any comparison. If you are not even able to demonstrate they are bad (and looking at the bottom note it's evident that you can't) why you're trying to do a comparison?

Whatever it is an internal comparative or a public one, it's poorly performed.

[Astech]

Is there any way to see what Malwarebytes detected from these 40k of samples?
I see and read a lot about Malwarebytes, most people say that it is able to detect malware that other products miss.

I don't want to take sides here, but everybody has a right to perform a test, I don't understand why every time Malwarebytes is tested we see the same situation, it caries the name Anti-Malware so I don't see anything wrong with putting it in the same testing group with the other programs with the same name.

As I see it all the top rated programs performed well, not to mention Avira, G Data, Kaspersky and Avast, they did very well considering that the samples were new, and as a user of one of those programs that makes me happy

I only wish that there are more tests out there, if there are more tests we would have more results to compare.

[emsisoft]

-{ Quote: "Bad one, my friend. This not the right answer. If you are not able to confirm the nature of samples you are going to test you can't release any comparison. If you are not even able to demonstrate they are bad (and looking at the bottom note it's evident that you can't) why you're trying to do a comparison?
" }-

In theoretical terms I fully agree with your logic, but in practice, the world looks a bit different. ;)

Are you working for an antivirus company? You know how AV companies decide if they add a signature for a file or not? Guess not..

OK, let me uncover an old industry secret and myth: There are not 5000 slaves in the cellar who test and reverse every single sample to make the best ever signature for detection. ;)

99+% of the malware signatures are created fully automated today. There is simply no chance to double the manpower of the analsis team every year. Companies that still rely mainly on manually added signatures are usually not the best in tests today. But more automated tasks automatically generate more false positives - that's not a secret.

Final question: If 10 top rated antivirus products detect more than 90% of these samples, would you believe them or would you believe vendors who's products detect less than 10% and who state that the 90% rest are not valid samples?

Detection changes every hour, but we would never drop 90% of the 40k detected samples from our database as FPs. We're talking about +-0.1% that might change within some weeks. That's absolutely normal in this industry and not a proof for bad software.

[MAOS]

Again, bad one my friend ;)

quoting from AV-Comparatives sorting procedures document:

-{ Quote: "
Samples are analyzed, using various tools (commercial tools, for example, but also tools used and maintained by the anti-virus community) in order to recognize known garbage or non-working samples. We also use several other static analyzers, PE parser, and so on, including our own in-house tools

(...)

All PE malware is analyzed by a sandbox developed by people working at AV-Comparatives and also by various commercial sandboxes, in order to exclude non-working samples and other garbage. Non-PE malware is also checked by some automated tools, but usually they need to be checked manually, as are some PE files that our sandbox was not able to categorize reliably.

(...)
" }-

Now, or at AV-Comparatives there are 5000 slaves working on those samples or you are clearly wrong, again ;)

[shah000]

I dont see a problem with the end result of this 'test'.
The top 4 make sense (from my experience).. i.e. Avira/A2-AM4/GData/Kaspersky

People complain about A2 False Positives..
I think the FPs are better than the misses.. but that means A2 is more suited to experienced users (those who care to see detected items instead of simply scan-remove)

The best thing is that A2 is made to work as a complimentary scanner.. so no harm having A2 on your system, especially when you can send possible FPs to another vendor for confirmation (if you think your main suite missed something that A2 caught).
Personally, i feel A2 AntiMalware 4 is a much better option than some 'overrated' antispyware softwares ;) but again.. no harm.. because you can use them all at the same time without issues.

[emsisoft]

-{ Quote: "Now, or at AV-Comparatives there are 5000 slaves working on those samples or you are clearly wrong, again ;)" }-

Please don't mix things. I wrote about signature creation, you about sample validation.

I never said that my test is objective. It's an subjective view on OUR samples. Samples that are worth being detected by our engine as well as by 10 other top antivirus products. For me and my test proof enough that I call them malware and include them in my test scenario. I've never said that my testset contains only valid working samples.

[MAOS]

-{ Quote: "Please don't mix things. I wrote about signature creation, you about sample validation.

I never said that my test is objective. It's an subjective view on OUR samples. Samples that are worth being detected by our engine as well as by 10 other top antivirus products. For me and my test proof enough that I call them malware and include them in my test scenario. I've never said that my testset contains only valid working samples." }-

The one who is mixing up things is you. I never talked about signature creation, I always talked about sample validation.

And this is the confirm you don't know what you are talking about: doing a test, releasing it online and then writing I've never said that my testset contains only valid working samples.

[NoIos]

-{ Quote: "That's all folks people :) Wilders users just have to read this to understand everything about the company" }-

:o
not a fortunate phrase

And since you have passed the line...sincerely...reading wilderssecurity...everyone can understand, using it's own mind, about almost all the security companies...and believe me what I have understood..is not that flattering for you guys..security experts...and your companies. So...

[LillyTown]

-{ Quote: "That's all folks people :) Wilders users just have to read this to understand everything about the company" }-
Cry-Baby

[MAOS]

-{ Quote: ":o
not a fortunate phrase

And since you have passed the line...sincerely...reading wilderssecurity...everyone can understand, using it's own mind, about almost all the security companies...and believe me what I have understood..is not that flattering for you guys..security experts...and your companies. So..." }-

Agreed, my sentence is a bit inappropriate. Deleted. I just don't like people who write "it's for internal use" and then release it online. So it is not for internal use, it's for marketing purpose. And if you want to do it for marketing purpose, at least do it in the right way and not so superficial.

That's all

[progress]

-{ Quote: "
I don't want to take sides here, but everybody has a right to perform a test, I don't understand why every time Malwarebytes is tested we see the same situation, it caries the name Anti-Malware so I don't see anything wrong with putting it in the same testing group with the other programs with the same name." }-

I think this is the major problem: Maybe it should be called Malwarebytes Anti-Rogueware because that's what it seems to be :)

[fcukdat]

Christian Mairoll [Emsi Software Team]

You just dont get it do you???

Peoples real world experiences of MBAM(&SAS) alike in no way reflect your contrived test results.

My NDA prevents me from discussing under the hood technology or detections routine in detail used by MBAM but then again why should we tell you why the software works a damn sight better than yours at removing active malware infections during real world usage.

Ok here's a little pointer:)

-{ Quote: "Avira is the clear 'winner' and MBAM is the clear loser because they were not able to manage signature creation for more than 39,000 malwares in the wild." }-

Why create how many 100's of strings to detect these samples and thus have to parse a larger DB when using a different approach means that a lot less DB space is needed to get the job done in the real world.

While your guys are busy stringing the usual suspects,our team is off hunting new malware that evades our DB in realife testing scenario's and writing new rules to attack them...this i why we are at the top of the malware killing food chain.

So if your sincerly interested in some real world comparison on detection and removal abilities here's some pointers for tests that would reflect a more truer picture of things.

Take SAS,MBAM and a2 only.

5 common infection scenario's

1)Find a few fake scanner pages and load up the FakeAlert rogues.

2)Hit the keygen sites(Type I+II) and run ther nice free keygens

3)Fire up P2P and grab yourself a worm infection.

4)Add application at my space and grab Koobface

5)Install codec for free pr0n.

Save image of each infection then test each of the 3 softwares against the infected image(s) and then publish your results.

All i can say is i hope you like coming bottom of the class but i guarantee you wont want to be lauding the results from realworld tests such as thoes:shifty:


Anyway with that can you pass a message onto your dev department as i found a real nasty side effect of your detection and removal engine when i last tested on a heavily infected machine(Keygen Type I).

You need to use whitelisting on critical system files.
Deleting system files that have been attacked by PE infectors only result in collapsing the OS and killing that install::)

It's ok that the system will need R&R because of the infection type but in deleting the critical files without warning that it will collapse the OS in no way gives a person chance to export(save) any pictures,media,personal documents etc :ouch: :'(

[Franklin]

Hmm, looks like I found the .1% A Squared misses and the 1.4% MBAM hits.

208517

208518

208519

[fcukdat]

lol Franklin,

To be fair the only way to compare would be to run the installer and get the realworld infection(Rogue install) then test both softwares against it.

That said the a2 tests remind of the old saying there are lies,damn lies and statistic's :shifty:

[Astech]

-{ Quote: "I think this is the major problem: Maybe it should be called Malwarebytes Anti-Rogueware because that's what it seems to be :)" }-

Yes, you see that would make more seance, and a lot less people would be confused.

Now we can all love or hate a-squared, one thing is for sure it does have an excellent detection ratio, False Positives sure they have a problem with them, but so does Avira, G Data, Avast.......

From all the talk I see here, everybody seems to be concerned about removing malware from the infected system, correct me if I'm wrong but it is ten times more important to prevent infections?
If you want to be good at preventing you have to detect, if you want to be good at detecting you need to think about both present and the past as well as the future, to make a long story short most of the vendors are getting their signature databases bigger and that is very smart because all the other systems are not yet advanced and functional enough to be the first line of defense........
You can't have a good Anti-Malware product with only a few hundred thousand signatures, ask malware researchers how many new malwares are being distributed/discovered every month, way too many:-\
High detection ratios come with the price, the price being paid are False Positives, but in most cases (if system files are not the problem), you can fix the FP's, in real circumstances it is very hard to clean a heavily infected system.

While all of you are arguing about this, there is a global pandemic of malware, it would be smart to put aside all the differences and work together for a change, only that way you can expect positive results:thumb:

[progress]

-{ Quote: "
To be fair the only way to compare would be to run the installer and get the realworld infection(Rogue install) then test both softwares against it.
" }-

I agree, but then you can also kick AV-Comparatives tests ::)

[Pedro]

-{ Quote: "
- It's clearly stated that this test doesn't say anything about a-squared's scanning capabilities because we used our own samples. In this view, Avira is the clear 'winner' and MBAM is the clear loser because they were not able to manage signature creation for more than 39,000 malwares in the wild. As said before, even I didn't know that MBAM are meant to detect ONLY active malware. Their website doesn't tell anything about it - it's sold like all other antivirus tools and as a first time visitor I'd believe that too. MBAM: Tell your customers the truth about your software abilities and we would start recommending it as a great addition to a-squared!
" }-
Hello,

I think it would be important to point out that, for the user, what you're saying is not that important. It would be important if, while executing said malware, MBAM would still not detect and block the threat.

Your test, and lets assume it's completely true, does not reflect 'real world scenarios'.
I'm not saying it's completely worthless, as i like reading on AV-Comparatives myself. But it does not tell the whole story, or depending on the products tested, not most of the story. Programs like Prevx for instance, or on the other extreme, MBAM, where it's developers explain why right-clicking a file to scan is not how it works best. Even BOClean, now gone.

This applies to AV's in general too, because they're not sleeping or watching other companies innovate, but to a less extent i think.

Testing isn't easy, but it's extra hard to ascertain what your "inhouse test" actually means, being a Marketing gimmick.

Don't take this as an aggressive post towards you or your company. It's my honest opinion.

[Leolas]

hey guys, I don't want to be like a "pacificator" but I believe people is bashing too much in this thread: emsisoft probably shoulnd't have shared this post or shouldn't have listed a-squared, and maybe Mike shouldn't have posted it, but:

also other companies do the same thing:


why nobody insults Prevx just because they have a comparison on their homepage -http://www.prevx.com/ ?


eset norton kaspersky and so on have done the same thing, but I haven't seen anyone saying that they are bad companies just because they did that



It's clearly written that it doesn't represent an objective assessment of the detection performance of a-squared Anti-Malware, because they are samples of emsisoft


the test is still good to see how the other softwares perform in an on-demand scan


I don't understand why you accuse Mike of being unprofessional: yes, they have an alliance with emsi, but if he hadn't posted the test, probably someone else would have done it soon.. so... dunno

Now, the only thing I didn't like is that Christian wasn't that nice with MBAM, but he has a point, the homepage of MBAM isn't clear about its functionalities, limits and so on. It seems like the only thing he wanted to do with this test, was pointing out MBAM's bad performance.. That's not what you wanted to do, Christian, was it? :P

[Kees1958]

FCUKDAT, FATDCUK, ADE

A Malware Researcher should have a library of malwares, a honeypot. What would be nice for a malware researcher is to proof it, let us show us the results of such a test.

Regards Kees

[Retadpuss]

I see nothing wrong with the test myself. The results seem to be very similar to those I have seen elsewhere. would like to have seen Prevx there as well.

[Saraceno]

-{ Quote: "hey guys, I don't want to be like a "pacificator" but I believe people is bashing too much in this thread: emsisoft probably shoulnd't have shared this post or shouldn't have listed a-squared, and maybe Mike shouldn't have posted it, but:

also other companies do the same thing:


why nobody insults Prevx just because they have a comparison on their homepage -http://www.prevx.com/ ?


eset norton kaspersky and so on have done the same thing, but I haven't seen anyone saying that they are bad companies just because they did that



It's clearly written that it doesn't represent an objective assessment of the detection performance of a-squared Anti-Malware, because they are samples of emsisoft


the test is still good to see how the other softwares perform in an on-demand scan


I don't understand why you accuse Mike of being unprofessional: yes, they have an alliance with emsi, but if he hadn't posted the test, probably someone else would have done it soon.. so... dunno

Now, the only thing I didn't like is that Christian wasn't that nice with MBAM, but he has a point, the homepage of MBAM isn't clear about its functionalities, limits and so on. It seems like the only thing he wanted to do with this test, was pointing out MBAM's bad performance.. That's not what you wanted to do, Christian, was it? :P" }-

Agree.

Use what you like, and what you feel comfortable with. If it's the free open source Moon Secure AV, so be it.

[fcukdat]

-{ Quote: "
You can't have a good Anti-Malware product with only a few hundred thousand signatures, ask malware researchers how many new malwares are being distributed/discovered every month, way too many:-\
" }-

Right without goin too much into detail but trying to make a simplified explaination this is an incorrect assumption because of what constitutes an individual signature and how it would perform in the real world.

There are many ways to detect and attack malware with different types of detections.
The figures i'm about to quote are pulled out of the air but will hopefully give a picture of what is going on.

Take a single trojan file and you can attack it by MD5 hash hit= 1 signature for that file.

The next time you download that file it has been tweaked and although it dose the same thing it has new MD 5 value (old signature now dose'nt work)

So next step up is to string the file so you are no longer dependent on 1 file= 1 signature ratio. A string might be worth say a 1000 MD5's depending on how long it is present in the target file.

So 1 string detection signature is a lot more valid then 1 MD HASH hit signature in the DB.

Now the fun begins because even string signatures have a limited shelf life as the original trojan gets tweaked some more to evade detection.

So this is where Heuristic's come into play8)
A good heuristic signature with no f/p generation is worth its weight in gold.

In certain cases that 1 Heuristic signature is worth every variation of that particular trojan from past,present and future variants yet to be even distributed.

Now simple maths says that 1 signature would take the place of potentially 100 of thousands of MD 5 hits or 100's of strings hits.


I hope this explains why the size of a softwares DB(number of signatures) is not proportional to its overall detection abilities:thumb:

[Pedro]

-{ Quote: "
Now, the only thing I didn't like is that Christian wasn't that nice with MBAM, but he has a point, the homepage of MBAM isn't clear about its functionalities, limits and so on. It seems like the only thing he wanted to do with this test, was pointing out MBAM's bad performance.. That's not what you wanted to do, Christian, was it? :P" }-
I'm not sure he actually has a point. Users expect protection, i don't see a static folder scan (with malware installers sort of speak) as indicative, or as a final word on how a product will ultimately protect the users.

[Saraceno]

-{ Quote: "lol Franklin,

To be fair the only way to compare would be to run the installer and get the realworld infection(Rogue install) then test both softwares against it.

That said the a2 tests remind of the old saying there are lies,damn lies and statistic's :shifty:" }-

Ade, you're now presenting MBAM, like the emsisoft test - saying your product is top of the malware food chain. Not that I disagree your product is exceptional, but people from other companies that get similar results to emsisoft (Norton, Avira, G-Data, especially Avast which is signature based), could say, serve up the detailed results.

This is like a pissing contest. Each product has its own strengths.

If everyone jumps on every company test, then no company will publish any results for fear of backlash. All we'll have is a few reviewers to rely on, that's it.

[jmonge]

i think a2square is an amzing product other than the many false positive which i dont mind A2 catches alot of nasties that also other misses and this is base on my own personal testing:) real malware that comes from danger websites(real situation)of all antimalware/antivirus signiture base shield this is the one i like(the only one)then my priority is Defensewall/malware defender;D

[Martijn2]

I have seen cases where MBAW did not detect a rogue scanner installed while A-squared did and removed it (visa versa!). I mean why are we bashing A-squared and/or MBAW/SAS while neither of them are the 'holy grail ', every product has his good and bad sides! Every test must be taken with a pinch of salt, Kingsoft/Rising for example work great in Asia, but score bad in our ' western' tests.

[NoIos]

-{ Quote: "I don't understand why you accuse Mike of being unprofessional: yes, they have an alliance with emsi, but if he hadn't posted the test, probably someone else would have done it soon.. so... dunno[/LIST]
" }-

I consider indeed unprofessional his action, but if I would like to be more gentle, respecting Mike N. for his great product ( since I don't know much about him ), I would call it ... unfortunate coincidence. If I was him...I would have let Emsisoft post the test...or like you say somebody else. I understand that because of this "alliance" he monitors the actions and progress of his partner...but hey life and forum experience command a different approach...better thinking before some actions. Nothing that bad though...things like these happen.

-{ Quote: "why nobody insults Prevx just because they have a comparison on their homepage -http://www.prevx.com/ ?" }-

You have missed some reading here. There is a thread about it.

-{ Quote: "It's clearly written that it doesn't represent an objective assessment of the detection performance of a-squared Anti-Malware, because they are samples of emsisoft" }-

I understand it and agree. But obviously others see it in a different way, which is also fair, they have theirs reasons and views...I accept it.

[Astech]

-{ Quote: "Right without goin too much into detail but trying to make a simplified explaination this is an incorrect assumption because of what constitutes an individual signature and how it would perform in the real world.

There are many ways to detect and attack malware with different types of detections.
The figures i'm about to quote are pulled out of the air but will hopefully give a picture of what is going on.

Take a single trojan file and you can attack it by MD5 hash hit= 1 signature for that file.

The next time you download that file it has been tweaked and although it dose the same thing it has new MD 5 value (old signature now dose'nt work)

So next step up is to string the file so you are no longer dependent on 1 file= 1 signature ratio. A string might be worth say a 1000 MD5's depending on how long it is present in the target file.

So 1 string detection signature is a lot more valid then 1 MD HASH hit signature in the DB.

Now the fun begins because even string signatures have a limited shelf life as the original trojan gets tweaked some more to evade detection.

So this is where Heuristic's come into play8)
A good heuristic signature with no f/p generation is worth its weight in gold.

In certain cases that 1 Heuristic signature is worth every variation of that particular trojan from past,present and future variants yet to be even distributed.

Now simple maths says that 1 signature would take the place of potentially 100 of thousands of MD 5 hits or 100's of strings hits.


I hope this explains why the size of a softwares DB(number of signatures) is not proportional to its overall detection abilities:thumb:" }-

I very well know that, but very few programs have heuristics that produce almost no FP's, so it is a win lose game just a matter who is assessing the the results.

Now I see you are a Malwarebytes researcher so I would like to ask you a simple question with no bad intentions: In your honest opinion, do you believe that Malwarebytes Anti-Malware can provide the same (or higher) level of protection then ,lets say, Avira, Kaspersky, Avast or G Data?

[fcukdat]

-{ Quote: "
Now I see you are a Malwarebytes researcher so I would like to ask you a simple question with no bad intentions: In your honest opinion, do you believe that Malwarebytes Anti-Malware can provide the same (or higher) level of protection then ,lets say, Avira, Kaspersky, Avast or G Data?" }-

You have just ask me to compare apples and orange's was that your intention???

Both have their respective strong points and both have their weak points.

Personally if someone needs a blacklist scanner to do their decision making for them then both are required.

Just food for though but if AV was all dominating then there would be no need for the *cleaners* nowadays but its very obvious that is not the case.

[Astech]

-{ Quote: "You have just ask me to compare apples and orange's was that your intention???

Both have their respective strong points and both have their weak points.

Personally if someone needs a blacklist scanner to do their decision making for them then both are required.

Just food for though but if AV was all dominating then there would be no need for the *cleaners* nowadays but its very obvious that is not the case." }-

So what you are saying is that Malwarebytes is a cleaner?

[andyman35]

-{ Quote: "lol Franklin,

To be fair the only way to compare would be to run the installer and get the realworld infection(Rogue install) then test both softwares against it.

That said the a2 tests remind of the old saying there are lies,damn lies and statistic's :shifty:" }-
Exactly! This malware needs to be executed in order for SAS and MBAM to show their prowess.Ok maybe Emsisoft made a genuine error in lumping these two in with a load of AVs for what was essentially an on demand scan of inert malware,if that's the case they should remove them from the results table until such time as they can run an appropriate test designed to show real-world performance.People will think far higher of them if they admit their mistake.

[andyman35]

-{ Quote: "I see nothing wrong with the test myself. The results seem to be very similar to those I have seen elsewhere. would like to have seen Prevx there as well." }-
The issue isn't the results that show AVs doing what they were designed to do it's the inclusion of products that shouldn't have been there.

[andyman35]

-{ Quote: "Ade, you're now presenting MBAM, like the emsisoft test - saying your product is top of the malware food chain. Not that I disagree your product is exceptional, but people from other companies that get similar results to emsisoft (Norton, Avira, G-Data, especially Avast which is signature based), could say, serve up the detailed results.

This is like a pissing contest. Each product has its own strengths.

If everyone jumps on every company test, then no company will publish any results for fear of backlash. All we'll have is a few reviewers to rely on, that's it." }-
No offence but I never saw anyone's PC get infected by a static folder of samples.

[fcukdat]

-{ Quote: "So what you are saying is that Malwarebytes is a cleaner?" }-

All botkillers are *cleaners* and most folks first encounter with MBAM or like software is when there pc's have been infected and they need software to come in and get rid of the mess.

Suffice to say MBAM's high clean up sucess rate is why it has gained popularity and a reputation that it has :)

[Retadpuss]

-{ Quote: "The issue isn't the results that show AVs doing what they were designed to do it's the inclusion of products that shouldn't have been there." }-

I think its the description of these utilities that has caused a problem. There are no AVs any more, everyone talkes about malware and AMs! A2 is called an AM, so are SAS and MBAM etc.

I guess you are saying SAS and MBAM are different sort of AM utilities. I hear lots about MBAM in particular being designed to detect what the mainstream AMs miss and so give users a better chance of getting towards 100% coverage.

I guess SAS and MBAM will detect the 0.1 - 1.3 % missed by A2 and Avira?

In some way, this makes sense as MBAM did find 1.4%, so if it is the case that it is designed to catch what the others miss then okay. - How do they know what the others miss? What if a user has one of the AMs that got 90% or lower? SAS and MBAM cant possibly help then as they get between 1.4 - 5.6%

It would be very easy to test SAS and MBAM on the samples missed by the top five AM - just to see if its true that they detect what the others dont.......

[IBK]

@emsi: plz recheck the files you used. I looked a bit at the md5 list you posted, and from a first check at least 2000 files are very probably junk (incl. known junk from the year 2006..), which are unprobable to be really worth detecting or present in a test-set. so, the files does not seem to be all that new, you can even check yourself some md5's on virustotals hashsearch and see that some files were uploaded already months ago (when also ikarus/emsi) did not detect them. where do those files come from? some few appear to be most probably manually unpacked and therefore most probably from some av vendor collection (or at least not real world user).
i would too suggest to remove mbam, sas, etc. and eventually re-do the test after having removed (at least the obvious) junk files.

-{ Quote: "So 1 string detection signature is a lot more valid then 1 MD HASH hit signature in the DB." }-
yeah, that's why i do not understand why some people think its a great thing if a vendor adds 5 million "signatures" in a week, but its product is unable to detect replicating stuff like Virut. number of signatures say nothing; well, maybe it says that if a vendor needs too have many signatures it has poor heuristic/generic detection :P

[TIfreak]

-{ Quote: "
Suffice to say MBAM's high clean up sucess rate is why it has gained popularity and a reputation that it has :)" }-

Ye you are really great etc. But besides this: you complain a lot here that testing MBAM this way is kind of unfair. So why do you want to make a headsup between a2 and MBAM by determining which program detects more running malwares? If I got it right a2 is designed to prevent malware from running on a computer. So that way of testing would be even more ********?

And honestly, I prefer malware not to be on my computer instead of killing active infections. The insecure feeling whether there really is no active pests running is just bad. Up to this thread I always thought MBAM is a really solid program which I was using a lot... but both the test and the way you argue here and you want to disgrace a2 changed this a lot. Please start to clearly show on your website that MBAM is an addition to existing malware prevention software and behave a bit different. Then I will maybe come back as a customer, thanks.

[fcukdat]

-{ Quote: "Ye you are really great etc. But besides this: you complain a lot here that testing MBAM this way is kind of unfair. So why do you want to make a headsup between a2 and MBAM by determining which program detects more running malwares? If I got it right a2 is designed to prevent malware from running on a computer. So that way of testing would be even more ********?

And honestly, I prefer malware not to be on my computer instead of killing active infections. The insecure feeling whether there really is no active pests running is just bad. Up to this thread I always thought MBAM is a really solid program which I was using a lot... but both the test and the way you argue here and you want to disgrace a2 changed this a lot. Please start to clearly show on your website that MBAM is an addition to existing malware prevention software and behave a bit different. Then I will maybe come back as a customer, thanks." }-

Welcome to Wilders :)

What has'nt been stated yet is that MBAM realtime Protection module is capable of blocking the active malware it removes.


So there is a correlation between high clean rate and high blocking rate although there is no way a2 tests are capable of reflecting this since none of the test files were active.

The problem is the static file recognition is not good but the software comes into its strenght when the malware/infection process is started or already active:thumb:

[Mattchu]

I have a simple question to ask...

If MBAM is really only truly effective when Malware is active and running on a system, why does it have the right-click scan built in?

Is there some `special` way that something sitting on the HDD can be detected.From what i gather Malware can only exist in 2 places, either on a drive or in memory.

The battle you AV/S/M guys are fighting is truly an immense task, good luck to all of you.

With respect to the test........well come on now.......!!

[subset]

-{ Quote: "I looked a bit at the md5 list you posted, and from a first check at least 2000 files are very probably junk (incl. known junk from the year 2006..), which are unprobable to be really worth detecting or present in a test-set." }-
Where can I find the MD5 list for the AV-C test-set? *puppy*

Cheers

[aigle]

-{ Quote: "yeah, that's why i do not understand why some people think its a great thing if a vendor adds 5 million "signatures" in a week, but its product is unable to detect replicating stuff like Virut. number of signatures say nothing; well, maybe it says that if a vendor needs too have many signatures it has poor heuristic/generic detection :P" }-
Totally understandable and I am always surprised on some poor souls who are happy that their favourite AV added so many and so much no of signatures in few days/ weeks etc, Moreover some people go one step ahead and even start new threads just to tell us that their favourite AV has added so many signatrures in few days, thinking that it,s going to change the AV world.

[Page42]

-{ Quote: "I expected that some vendors who are not happy with the results would allege that our test is fake. To all of them: Don't try to terrorize us with lawyer threats, better tell us what else information we should present to make our testing methodology 100% transparent." }-
Gee, I wonder who is threatening you with lawyers? ;D I've seen it here before. It's SOP for one guy. Everytime his software draws a poor review, out come the lawyer threats.

[Page42]

Someone asked, if the programs that did poorly on the test are really only truly effective when malware is active and running on a system, why do they have right-click scanning built in? Excellent question.

[Dr33]

the test is a total scam for normal costumers but many other companies do the same, if you want to test your samples vs other vendors you should not include your software in the list period.

is like you take a math test and you already know all the answers :wacko:

the story will be diferent if you grab samples from diferent countries and test against those products before sending it to your labs i think it will be more acurated

then we will know how well A-Squared and other companies will handle Brazilian Malware Chinese Malware and Russian Malware :lurking:

[littlebits]

Thank you very much for this test Emsi Software!;)
Some may criticize this test but if you look at all of the other testing sites, the figures are all about the same. The only difference is you tested other programs that suppose to be Anti-Malware products some of which have never been put to the test before. The only real shock to myself was eScan :o , it did better than what I believe it could do.

SUPERAntiSpyware is excellent for spyware, adware and tracking cookies even though it didn't do so well as a full malware solution, I believe it does its intended job.

It was nice to see Comodo do so well also for a new product almost as good as Nod32 already.

For MBAM, I'm not surprised at all, I have put it to the test several times on many systems with all types of samples and it is really nothing more than a rogue software removal tool. Yes it is a useful product to many users (as a free product), but many believe it to be a complete solution when they buy it and I feel sorry for them when they get multiple infections and have to turn to other products for help. You have to think most of online users are in the novice category, they don't read forums, they just read the products home page or the download site's description and then download and sometimes buy the product. It really isn't fare to them because the description of MBAM doesn't tell them important facts. Just read the description for yourself and pretend that you are a novice user.

Good Day everyone.:)

[RubbeR DuckY]

Unfortunately some people will never learn. Personally, I am done with this topic. Google is the answer to all questions asked in this thread. I will be the 'bigger man' and step away from this thread. Repeated attempts to contact anyone at Emsisoft have failed, funny isn't it.

Mike Nash, it was a pleasure to meet you. Hopefully our next encounter will be a bit more friendly -- I did not mean to put you on the spot, I just wanted to understand the motives.

As for Emsisoft, personally, I wish you the best. I clearly see losing business to us is affecting you to the point that you need to do a test where you come out at the top and then put in very fine print that the results do not count for your product. Why not take the normal route and bundle a toolbar like others do. Regardless, business is business and you clearly showed me that.

To anyone I offended, I apologize. If you have any questions regarding Malwarebytes' Anti-Malware you are always more than welcome at our forums or to contact me via private message here.

I'm going to go back to work on the product -- I urge the others to do the same. Or, you can continue to argue, that works too I suppose, but I will not be part of it.

[arran]

When reading thru this thread I find it quite Entertaining, as for the test results
who cares which product is best and whether or not the results are fake or genuine. because I have a much better strategy which is preventing infection from happening in the first place, so I don't need to rely on products to clean out the malware after the damage is already done and I need to worry if my av or antispyware program will detect all the malware I have. Because I prevent infection from happening in the first place.

[emsisoft]

-{ Quote: "Repeated attempts to contact anyone at Emsisoft have failed, funny isn't it." }-

Where did you send them to? My email is stated on our website: emsi at emsisoft dot com

Try to avoid too many bad words, the spam filter might drop it otherwise. ;)

Didn't find a PM from you here on wilders too.

Btw. will a google search tell me more that your product isn't a full malware scanner, but a complementary tool?

[RubbeR DuckY]

Christian, I sent it to info@, and I assure you there were no bad words.

Our product is a full anti-malware tool that scans for Trojans, Worms, Spyware, Adware, Dialers, Rogues, and a limited amount of viruses. There are a multitude of posts on our forums stating that we are a complementary tool to any anti-virus, but not another anti-malware. We encourage a layered approach.

Regardless, let's talk about it via e-mail so I don't have to check this thread anymore.

[emsisoft]

Ok, found your mail in the junk folder. Reply is on the way..

[LagerX]

-{ Quote: "Thank you very much for this test Emsi Software!;)
Some may criticize this test but if you look at all of the other testing sites, the figures are all about the same. The only difference is you tested other programs that suppose to be Anti-Malware products some of which have never been put to the test before. The only real shock to myself was eScan :o , it did better than what I believe it could do.

SUPERAntiSpyware is excellent for spyware, adware and tracking cookies even though it didn't do so well as a full malware solution, I believe it does its intended job.

It was nice to see Comodo do so well also for a new product almost as good as Nod32 already.

For MBAM, I'm not surprised at all, I have put it to the test several times on many systems with all types of samples and it is really nothing more than a rogue software removal tool. Yes it is a useful product to many users (as a free product), but many believe it to be a complete solution when they buy it and I feel sorry for them when they get multiple infections and have to turn to other products for help. You have to think most of online users are in the novice category, they don't read forums, they just read the products home page or the download site's description and then download and sometimes buy the product. It really isn't fare to them because the description of MBAM doesn't tell them important facts. Just read the description for yourself and pretend that you are a novice user.

Good Day everyone.:)" }-

I can't agree. MBAM is a very nice little application when computer is infected. If you manage to install and update it, MBAM does its job very well!
I've cleaned many computers with it, where other tools are useless. Especially with it's quick scan. I've dealt much with that "gaop" named rootkit file(s) and MBAM has always been successful. I don't have time to download ~20mb setups and do full scan. Those product's "quick scan" features aren't always as good as MBAM, because MBAM "knows" where to look at malware.

Just my opinion:)

[progress]

-{ Quote: "Someone asked, if the programs that did poorly on the test are really only truly effective when malware is active and running on a system, why do they have right-click scanning built in? Excellent question." }-

:thumb:

[hammerman]

-{ Quote: "For sure there are FPs, daily, but they're fixed quickly." }-
How quick is quick. I reported and submitted a Virut FP a month ago and today's scan result still shows Virut.

http://img525.imageshack.us/img525/772/20090502080052.jpg

[emsisoft]

Could you please submit that file once again? In the scanner, there is a "Report false alert" item in the right click menu on the detected item. That's the best way to submit and ensure that it will be fixed asap.

99% of all reported false alerts are fixed within 24 hours.

[arran]

lol thats a file part of EQS, I find it kinda funny.

"you are urgently advised to block this program" haha lol

[Retadpuss]

-{ Quote: "It is a shame that a reputable company such as Emsisoft would be so desperate to suck money out of their customers that they would resort to discrediting other reputable companies with a test clearly suited to put a-squared in first place. Just business I guess right? What's next, a toolbar? I heard that's a great business move too.

Personally, I'm touched. As the owner of a company that I know will outperform a-squared anti-malware in a live test I am just sitting here with a huge grin knowing they had to resort to this.

Customer happiness > Money in our opinion. Maybe not theirs." }-

Is this you? Is that one of your malware researchers next to you?

http://www.facebook.com/people/Marcin-Kleczynski/618993310

Its laughable that you say use Google to see how good MBAM is, when the fact is every test that has ever been performed which includes MBAM has the same results the one by Emsi.

Im confident there are many people who would like to take you up on your challenge and test MBAM against A2 in a live test. There are probably people here from independant testing groups who can conduct such a test.

Could anyone here who is from such a group / organisation let us know if a test can be performed. It would be good to see the results.

[Someone]

-{ Quote: "when the fact is every test that has ever been performed which includes MBAM has the same results the one by Emsi." }-
What about this test (http://www.techsupportalert.com/best-free-adware-spyware-scumware-remover.htm)?

[BrendanK.]

Can the bashing please stop. It is really not needed >:(

But just to say something really quick:

Just because this test shows some products in a bad light, doesn't mean they are bad at all.

And Emsisoft here is some food for thought ;) This is why people use MBAM :) I'm sure you understand that your product has it's own strengths and weaknesses but so does every product. It doesn't mean they are a complete write off.

http://img515.imageshack.us/img515/5278/42131022.png

[Smokey]

-{ Quote: "
Im confident there are many people who would like to take you up on your challenge and test MBAM against A2 in a live test. There are probably people here from independant testing groups who can conduct such a test.

Could anyone here who is from such a group / organisation let us know if a test can be performed. It would be good to see the results." }-
Well, as long you finance such (absurd) test, I am sure you will find somebody to perform the test desired by you. Keep in mind that no serious, acknowledged tester will accept your test mandate, even not for $$$.

<S>

[Retadpuss]

-{ Quote: "What about this test (http://www.techsupportalert.com/best-free-adware-spyware-scumware-remover.htm)?" }-
That is not a propper test - read it:

"I went hunting around the seedier side of Hong Kong websites, picking up as many infections as I could find in a 30 minute surfing session, and then seeing how today’s anti-malware software coped"

A real test is one with a known and large sample of malware (real world malware) conducted with a published methodology, like those at http://malwareresearchgroup.com/ - who interestingly have conducted live infection tests.

[Someone]

-{ Quote: "That is not a propper test - read it:

"I went hunting around the seedier side of Hong Kong websites, picking up as many infections as I could find in a 30 minute surfing session, and then seeing how today’s anti-malware software coped"

A real test is one with a known and large sample of malware (real world malware) conducted with a published methodology, like those at http://malwareresearchgroup.com/ - who interestingly have conducted live infection tests." }-
How much more "real world" do you want? The guy acted like what a normal user would do; browse the web then try to clean the infection.

[TIfreak]

-{ Quote: "Personally, I am done with this topic. I will be the 'bigger man' and step away from this thread. " }-

Dude can you be less theatrical? MBAM employees spamming this thread all the time and crying like a baby about the unfair world is not really the behaviour of a "bigger man" and its fellows. You state your product is overall better than a2. So please prove it instead of showing the mentioned behaviour?

[Mattchu]

Still no answer to my question...!!

WHY if MBAM is only truly effective when malware is active and running on a system is the right click scan option there.

I would prefer not to execute in the first place.Perhaps this is why a couple of `on-demand` scanners is preferable...

Not looking for an argument just clarification, does it utilise it`s advanced heuristcs on dormant files/folders?

[Jadda]

-{ Quote: "Funny to see all the bashing early in the morning, wilders will never change.. ;)" }-

Did you expect anything else? You guys asked for it. I love your attitude as a professional company too.

[Astech]

This thread is becoming a flame war, people take a deep breath, Emsi Softaware has done nothing that others didn't do before.

We had a chance to hear from the Malwarebytes representatives, it is clear to everybody now that MBAM is a tool for clean up, end of story8)

Stop acting like kids if you want to sound like professionals;)

[slangen]

i love this thread, though couldnt get past page 3....

thanks guys for 'Reality Security' :thumb:

[andyman35]

-{ Quote: "How much more "real world" do you want? The guy acted like what a normal user would do; browse the web then try to clean the infection." }-
Excellent point:thumb: In the real world average web users don't act like malware researchers.

[Saraceno]

To me, right-click scan is important.

And it must be to plenty of others. That's why plenty of guys here upload suspicious files to virustotal (and others), to see if a file is malicious before it is installed. They don't install a file first, then run the scan.

I agree with almost all arguments here.

SAS and MBAM shouldn't be included in right-click scan tests, as they work on active infections, but at the same time, these products should clearly describe their role in layered security and use as additional tools.

And for every test you find showing a product as being great, you'll find another test saying it sucks. :) 1:1 ratio.

Unless you're a company like Symantec/Norton, and nothing can stop the cash from rolling in. ;)

[Makav3l1]

I understand that MBAM and SAS don't like this test. However, a lot of their users use them as on-demand scanners so this kind of test appears applicable. Aside from emsisoft, which is expected to be on top with it's own samples, the other products results seem to follow the norm.

[snapdragin]

-{ Quote: "And for every test you find showing a product as being great, you'll find another test saying it sucks. :) 1:1 ratio." }-Exactly, and the same thing happens with the replies to those tests. Around and around it goes. I really don't see where this thread is progressing, so we're going to give it a rest.

Thread closed.