MRG Effitas 360 Assessment & Certification Programme Q 1 201 7

Source:

https://www.mrg-effitas.com/wp-content/uploads/2017/05/MRG-Effitas-360-Assessment-2017-Q1_wm.pdf

17 Applications Tested

386 In-the-Wild malware samples used

Interesting results ...more in the link above

 

Indeed some interesting results...

 

Oh, ouch! Avira must be celebrating, however. I feel for Malwarebytes right now.

 

MRG used Edge as the browser for this test. Unless MRG specifically disabled Win 10 native SmartScreen and I see no reason why they would since its enabled by default, the argument that SS was disabled for Windows Defender poor test scores is a moot point. The default setting for Edge is to use native SmartScreen. @Zoltan_MRG comments on this?

Eset and HMP scored 100% in ransomware protection.

Interesting comments by Kaspersky as to reasoning for their lower test scores.

 

Wow!

 

I see that but I question whether it's not HitmanPro Alert which has all the real-time modules. HitmanPro is the adjunct on-demand scanner. Which is it?

 

HitmanPro (on-demand scanner):
Page 7: "Security Applications Tested [...] SurfRight HitmanPro 3.7.15 - Build 281"

 

I believe HMP-A protections are now incorporated into the paid vers. of HMP?

 

These MRG tests ...
You can call them a lot of things, yet "interesting" are not one of the words that pops up in my mind.

As everyone remember from the last MRG test, then MRG told they disable SmartScreen and UAC in all tests.

Not very smart to disable URL-filter in Edge, when they (and everybody else, rightfully) praise having URL-filtering in the kill-chain.
Not very smart to disable UAC, since first of all it enables samples to elevate freely. And it also disables SmartUAC which in Windows 10 triggers a deep scan from WD of anything that tries to elevate privileges.

Remembering a little further back, MRG wrote they disables automatic sample submission.
Not very smart to disable automatic sample submission in WD, since that also disables the Block at First Sight feature in WD.

Finally I notice that graphs and written test results do not match.
In the Financial Malware section we can read that out of a sample set of 67 samples, Microsoft missed 0 samples, 4 samples blocked by behavioral blocking and 63 samples where auto blocked.
Yet the graph claim that there was missed samples ?

I must say that these MRG tests seems to be a complete mess, as already discussed in the last MRG thread some months ago.

 

nope

 

+1 Great post

 

Another strange thing of this test: Avast and AVG share the same identical engine. So why AVG scores a better result?

 

+2

 

because their pro-active technologies.

 

Thanks, mood, clearly it can be ASSUMED I did not click and load the link, I only looked at it. This would pertain to detection via HitmanPro, then, not protection via the Alert.

 

Also appears MRG changed their scoring criteria in that any detection other than auto-block was considered a failure. That is, user determined action via behavior detection was a failure. Explains Symantec's low scores.

 

If you look at page 11 Financial Malware of the report you will notice that Webroot has a red portion at the top of their column which indicates it failed on 2 or 3 items yet the score breakdown at the bottom gives a score of 100% with no failures. Not the first time we have seen this 'mistake' with Webroot in MRG 'tests! I wonder what the chart will show when it's corrected

 

With smart-screen disabled, and other security features of WIN10 disabled, this "test" is not worth a penny, or a cent.

 

https://www.wilderssecurity.com/thr...-windows-10-needs.383448/page-25#post-2674261

 

Hence my following post on the other thread.

Anyway now, there is no standalone WD, it is part of a wider mechanism (aka Windows Built-in Security, all grouped into WD Security Center) ; so test it with all features it works with or don't include it in the test or test it only with other single scanners.

 

guest,avast and AVG both share each others cloud and IDP engines from what we know.I have raised this question of variation in detection to avast anyway.

My queries here are:

1)Was there any time gap between testing of a URL on each machine with a product? If so the test is unfair even a half hour time frame is enough for a AV cloud to flag a previously undetected malware as bad.

2)I read their criteria where they explain "missed/bypassed" okay so what if the sample was anti-vm as most ransom malware is now days and did nothing to the system anyway.They do support anti-vm samples that won't do dirty work in a VM.If you ask me this is the problem if the file doesn't do its dirty work BB cant stop it.Kaspersky has made a argument about 2 malware files that were broken and did no harm.
In the Financial Malware section we can read that out of a sample set of 67 samples, Microsoft missed 0 samples, 4 samples blocked by behavioral blocking and 63 samples where auto blocked.
Yet the graph claim that there was missed samples ?

 

they didn't precised this.

indeed.

the report graphs has many (voluntary?) issues. People just look the graph , they barely read the small description under...

 

Yup, seems to be the same old sorry. Certain fans-boys can't deal with the ugly and messy truth and come up with the same lame excuses again. A big thanks to MRG for exposing this stuff, so Win Defender is still crap even on Win 10, and Malwarebytes really should stop promoting itself as an AV replacement.