Malware Research Group #23 Test

http://malwareresearchgroup.com/?page_id=2

You can download the test on that page. Here are the results anyway...


~~ image of results table removed ~~

~~ The copyright within the PDF document is very specific about not publishing or reproducing its contents. ~~

 

Feel lucky you didn't decide to post this over at ___ (another forum).

Anyway, only one major surprise to me, everything else looks about right (whether they're credible or not the results are believable IMO).
...well ..with the exception of a certain one anyway, and no I don't/won't use it.

 

Crystal clear. You won't be banned for copyright reasons, that's for sure.

 

There is a line in the pdf which shows some lack of knowledge.
"... Oasis components are not used in static scanning."
That's wrong. Oasis is used by OA for static scanning.
And as the tests were performed with Internet access, OA used Oasis for sure.

There may be no difference related to the scan results, if only malware is scanned, but OA uses its own whitelist during static scanning to prevent FPs by the AV engine, in fact Oasis overrules AV detections.

Cheers

 

We are totally aware of that, but as NO false positives were used in this test it made no difference.

Internet access was active yes, but all programs were tested at the same time and giving them exactly the same testing conditions was necessary.

Regards,

Sveta

 

Hi Sveta,
Why wasn't Hitman used in this test? Its sole function is an on-deman scanning, I would think it would be very fitting among the products you tested.
Thanks!

 

Testing cloud based Anti-Malware applications requires a more complex methodology.

By their very nature, many cloud based applications are more intelligent and can become aware they are being tested. For instance, if we were to scan 250K samples with uniform file names MRG_TEST_01 etc sitting in a folder on the desktop of a single system a vendor could be aware we are testing malware!

We have devised a methodology for countering any such issues, but details of this are private – but, we can say that it would not have been compatible with this test.

Regards,
Sveta

 

MRG,
How does your test differ from AV-comparitives and VB100? Asquared received a less than stellar performance lately on I believe the VB100. On your test it received top honors.

 

I think that MRG did not test for false positive and both VB100 and AV-Comparatives do.

 

Bit surprisded here too. G-Data and Avira near the top. Not surprising. Only A-squared is. I would also love to know how PrevX and Hitman Pro would do on this test. Aren't Mcaffee and Norton in the cloud too?

 

Well as far as I know AVC didn't test A-Squared and they failed only one part of VB100 test. Please read the VB100 report as see where exactly A-Squared/Ikarus failed.

Our On Demand tests are strictly about detection, we do not use false positives, therefore we give out awards to those with highest detection rate.

 

Thanks for the quick reply.

 

How does a2 anti-malware have a highest detection rate? even higher than G-Data (Bitdefender and Avast engine) and Avira (a lot of users).

 

A note: FP's didn't count in this test.

Reasons:
1. A-Squared also uses two engines (Emsisoft and Ikarus)
2. A-Squared has an awful lot of signatures, look at the signature updates.
3. Ikarus has really aggresive heuristics, and because FP's don't count in this test it will get a high detection rate.

 

So would I. Prevx declines to be tested. They say that tests are NG when it comes to Prevx. Hmmmmm........

 

So the "Flag-all-files-AV" is the winner, I see ...

 

No, "detect-most-files-AV" is the winner, all files malicious

 

Yes, so if I write an "AV" that will flag all files as malicious then I surely win, right? Sounds like your methodology is kinda broken.

 

Flagging files is one thing, detecting verified samples is another. Don't mix those two things, I was very clear in my previous post.

 

Yes - Doktornotor AV will get the golden award

 

How's it different in your case? You say you only test w/ malware. So, if I flag everything, I can't miss anything and I win. Of course such AV is totally useless, but definitely a winner.

 

Well very different mate, you can flag whatever you want and detect it, but that doesn't mean that you product is good.
I my book its simple, 250+k of samples (real malware), the application that detects the most, wins.

 

Well, no offense - but, logic wasn't your forte back at school, right? Once again, the test looks heavily broken, since it doesn't require any real intelligence to pass it. You only need one and exactly one universal signature which matches all files. And yeah, it doesn't mean the product is good - then, why are you using such flawed methodology?

 

Sort of like current HIPS tests. You sit a user in front of a computer and tell him that all the popups will be malware. Fails must be because the application doesn't generate quite enough popups, so you have quiet modes for users and robust modes for testing. I think we did that with monkeys and food when I was in school.

 

The methodology is not flawed And how are you going to create one universal signature to detect various types of malware? I would really like to know, as well as the whole internet security industry for that matter...