Interesting Emsisoft Tests

I'm sorry for bringing up sandboxie,in this thread. I just had it on my mind at the time.

Back on topic.

Regarding how legitimate the test results really are I chalenge the venders posting in this thread to have their products tested at av comparatives then we will all know for sure.

 

They have already been tested by independent groups And they did very well ranking in the top spot if not in the top 3. As for MBAM and SAS, I'm not sure of any tests. But I do know they protect me like nothing else can (of course they are a part of my layered approach and are not my only app which I love and could not live without, but for the spot they hold in my layers of security they are the best) They detect the stuff others miss, so for them to detect every possible malware sample is just plain stupid. I like MBAM and SAS how they are regardless of any test.

My faith is loyal

 

Personally I'm always surprised to read all this FP from A-Squared. I've been used it free for regular on demand scans from many years and I had very few FP. I use A-Squared as control scan of my system, and it's very powerful and reliable.

 

Agreed. I have had a few false positives from time to time, but the same can be said for the other programs I have used.

 

Actually that test said it is "NEW" Samples so maybe Super Antispyware isn't good with "NEW" Malware samples but it has a good detection rate for all other malware??

 

I don't think some people read the whole page - just looked at the graph and then saw red.

From my understanding, Emsisoft already had the samples so it's a given they would have added these.

Programs like Avira, Kaspersky, G-Data, and Avast to me were the clear winners. They were able to detect files that Emsisoft already had and knew were malware.

1. Avira
2. G-Data
3. Kaspersky
4. Avast
5. Norton
? Emsisoft (a-squared)

And to say right-clicking on a folder is a poor test, I think you have to realise it for what it is, just an on-demand test. People here are always right-clicking and scanning files. I right-click and scan files recovered from sandboxie. I right-click and scan through a bunch of files I've copied on to USB. I don't launch these programs, I'm right-clicking and scanning the files.

So right-click and scan, is still an important test, not the most important, but still, it's important.

If the test didn't state it was a simple right-click and scan, then I can see people getting upset.

I'm a big fan of almost all the products in the test, including SAS, MBAM and Emsisoft. But I'm not going to swear my loyalty to a product by carving the company name on my chest and going bananas when someone says otherwise.

Just relax, if the CEO Christian isn't aware of MBAM or SAS and their capabilities, then he's missing out on observing some valuable security programs.

If you bash on a company, then you're only displaying the same behaviour as you're alleging they displayed.

 

Whatever, do you really believe to a company who release a comparative test and they do not have even tested samples they are using for testing? And they are so smart to add a note saying that "after in depth analysis some samples are not considered malicious"?

And I will not comment about their superb Mamutu...

 

You've proven my point about bashing a product, 'superb Mamutu'.

Please release your thorough tests of Mamutu, which a lot of people here use, can't wait to read them.

Also, I'm no AV expert, but I remember reading how long it takes Dr Web researchers to analyse some files. So I'm gathering, it does take some time determining whether a file is completely clean.

 

Christian,

First congrats with the OA alliance. Secondly it is a perfect explanation that Ikarus scored high in FP's due to the lack of international operation. Consumer to consumer communication exactly describes your reputation, so you need to take other measurements, like publishing those low FP reviews on your website also.

With themed messaging (this test and new low FP's messages) you work on your company's and product's identity. It really pays off when you align your communication with the release calendar of your products. Simply asking people to change their minds won't work.

Publishing a test which is completely dominated by your own test set, is not an independant comparison, because the priciples on which it is performed, are benifial to A2's developers team. To other vendors it would trigger the same reaction as the stupid test of Mamuto on Matousec. I would not impose something on others, which you do not want to impose on yourself.


Regards Kees

 

If analyzing malicious softwares takes a lot of time, then it doesn't allow you to do and release comparative testing and then fix them because some malicious softwares were not really malicious after analysis.

When you've done your research and you're sure about samples you are going to test then you can do comparative testing.

By doing so you're only showing your absent skills.

About Mamutu: what you're expecting from a totally user mode software which applies its hooks only to Win32 subsystem API? Yes, it can gather behaviors from executed software.

Yet, it can be easily bypassed and many malwares are able to analyze themselves and fix API hooks or totally bypass them by calling directly the dispatcher.

Everyone in the security industry knows exactly how bad can be using user mode API hooking.

 

I will never get tired repeating that the combination a-squared + Ikarus is one of the best I have ever seen in many years. Using it on every day conditions as an on demand scanner...the last 5 months has missed only a few malware in downloaded files. Some FPs yes...but the rate is really small.

Now, I think it was an error that Mike Nash came here to inform us about the test. It's not bad to provide informations and personally thank him for his post, but he should act and think more professionally before posting. Most of us already knew about the Emsisoft + Tall Emu "alliance", so it didn't seem that elegant see him posting about it. Of course he's a free man...he can do whatever he likes.

MBAM and SAS have my respect too. They do a fine job. The only problems I had with MBAM was some false positives with fake drivers and SAS...although I had reported some files many times as false positives...they got flagged by their team as non malware months later.
For the rest they work great. I agree with all those who say that MBAM and SAS have a different point of view about security, so normally they don't fit in a regular AV test.

The problem with MBAM, SAS and a-squared is that they have not managed till now to convince me that I need their real time protection...having Avira or Avast installed + a firewall with hips. All 3 programs are great as on-demand scanners.

 

And, most important, just an internal comparison. No more, no less.

How about removal capabilities? Apparently not part of your test, nevertheless important. Anyway, to me.

<S>

 

MAOS, remember, most people using computers/laptops aren't in the security industry. You're talking about 1 per cent or less of users.

A product that appeals to the 1 per cent might cripple a novice user. So different products are needed for different users.

 

I expected that some vendors who are not happy with the results would allege that our test is fake. To all of them: Don't try to terrorize us with lawyer threats, better tell us what else information we should present to make our testing methodology 100% transparent.

Here is the hash list of the tested samples:
http://www.emsisoft.com/en/software/scanner/antivirustest_a2samples_200904_hashes.txt

And here is the list of malware names that a-squared detected:
http://www.emsisoft.com/en/software/scanner/antivirustest_a2samples_200904_names.txt (that list contains a bit more lines than scanned files because some samples were packed and the scan log contained 2 or more lines per file)

We did not test every one of the 40,000 samples if they are still able to run. No tester can do that for that huge amount of malware. But remember: These samples were worth to be included in our signatures, so you can presume that they're not just harmless ascii files. The majority of all antivirus programs detected them too.

Interesting note: Even if you pick only the major spread ones (conficker, etc), they're much more samples than MBAM detected at all (~550 of 40k).

 

You asked me for an explanation, and I've given a technical one to you. I know that 99% of people will not understand that, this is why I've just written in simple words that Mamutu isn't a good software and it can be easily bypassed.

This one should be easier to understand

 

Bad one, my friend. This not the right answer. If you are not able to confirm the nature of samples you are going to test you can't release any comparison. If you are not even able to demonstrate they are bad (and looking at the bottom note it's evident that you can't) why you're trying to do a comparison?

Whatever it is an internal comparative or a public one, it's poorly performed.

 

Is there any way to see what Malwarebytes detected from these 40k of samples?
I see and read a lot about Malwarebytes, most people say that it is able to detect malware that other products miss.

I don't want to take sides here, but everybody has a right to perform a test, I don't understand why every time Malwarebytes is tested we see the same situation, it caries the name Anti-Malware so I don't see anything wrong with putting it in the same testing group with the other programs with the same name.

As I see it all the top rated programs performed well, not to mention Avira, G Data, Kaspersky and Avast, they did very well considering that the samples were new, and as a user of one of those programs that makes me happy

I only wish that there are more tests out there, if there are more tests we would have more results to compare.

 

In theoretical terms I fully agree with your logic, but in practice, the world looks a bit different.

Are you working for an antivirus company? You know how AV companies decide if they add a signature for a file or not? Guess not..

OK, let me uncover an old industry secret and myth: There are not 5000 slaves in the cellar who test and reverse every single sample to make the best ever signature for detection.

99+% of the malware signatures are created fully automated today. There is simply no chance to double the manpower of the analsis team every year. Companies that still rely mainly on manually added signatures are usually not the best in tests today. But more automated tasks automatically generate more false positives - that's not a secret.

Final question: If 10 top rated antivirus products detect more than 90% of these samples, would you believe them or would you believe vendors who's products detect less than 10% and who state that the 90% rest are not valid samples?

Detection changes every hour, but we would never drop 90% of the 40k detected samples from our database as FPs. We're talking about +-0.1% that might change within some weeks. That's absolutely normal in this industry and not a proof for bad software.

 

Again, bad one my friend

quoting from AV-Comparatives sorting procedures document:

Now, or at AV-Comparatives there are 5000 slaves working on those samples or you are clearly wrong, again

 

I dont see a problem with the end result of this 'test'.
The top 4 make sense (from my experience).. i.e. Avira/A2-AM4/GData/Kaspersky

People complain about A2 False Positives..
I think the FPs are better than the misses.. but that means A2 is more suited to experienced users (those who care to see detected items instead of simply scan-remove)

The best thing is that A2 is made to work as a complimentary scanner.. so no harm having A2 on your system, especially when you can send possible FPs to another vendor for confirmation (if you think your main suite missed something that A2 caught).
Personally, i feel A2 AntiMalware 4 is a much better option than some 'overrated' antispyware softwares but again.. no harm.. because you can use them all at the same time without issues.

 

Please don't mix things. I wrote about signature creation, you about sample validation.

I never said that my test is objective. It's an subjective view on OUR samples. Samples that are worth being detected by our engine as well as by 10 other top antivirus products. For me and my test proof enough that I call them malware and include them in my test scenario. I've never said that my testset contains only valid working samples.

 

The one who is mixing up things is you. I never talked about signature creation, I always talked about sample validation.

And this is the confirm you don't know what you are talking about: doing a test, releasing it online and then writing I've never said that my testset contains only valid working samples.

 

not a fortunate phrase

And since you have passed the line...sincerely...reading wilderssecurity...everyone can understand, using it's own mind, about almost all the security companies...and believe me what I have understood..is not that flattering for you guys..security experts...and your companies. So...

 

Cry-Baby

 

Agreed, my sentence is a bit inappropriate. Deleted. I just don't like people who write "it's for internal use" and then release it online. So it is not for internal use, it's for marketing purpose. And if you want to do it for marketing purpose, at least do it in the right way and not so superficial.

That's all