MRG Effitas 360 Assessment & Certification Programme Q2 2017

So you're saying that cloud scanning and local behavioral monitoring was not used? That should explain why WD performed so badly, missing 82 samples is not acceptable. And the reason I was surprised about SmartScreen (SS) is because according to my theory it should be able to block 100% of all malware, since it's a black and white-list. So you don't even need an AV (for blocking) with SS enabled.

But apparently, MRG reported it as a fail if SS didn't clearly identified that you're dealing with malware. Another interesting to note, is that apparently SS was able to catch 63 samples that WD could not, without the white-list. So SS and WD are not using the same blacklist, or perhaps it was because of the cloud.

 

Win 10 native SmartScreen only monitors direct program startup. It won't monitor anything started from a shell for example.

-EDIT- When native SmartScreen is employed in conjunction with WD, it is assumed that WD is the one detecting process startup and then it consults with native SmartScreen to perform the rep scanning of the process.

The main thing to note is WD's max. effectiveness is only achievable on Win 10 since all the following security components are employed; AMSI for memory script based scanning, native SmartScreen, and the ELAM driver that protects WD from malware disabling and tampering. Additionally unless a Microsoft browser is used, there is no browser based security e.g. phishing, exploit, drive-by download, etc. since that is being deployed by the browser based ver. of SmartScreen.

The main current issue with Win 10 native SmartScreen is that it is a separate process, not protected, and can be easily disabled by malware. It doesn't even have AppContainer protection and runs at Medium integrity level.

 

Well, not everything is clear to me, but I'm guessing that if SS can catch malware via the cloud, so can WD if the cloud is enabled of course. But that would still mean that it missed 19 samples which is not a good score. But what definitely is not clear is what MRG means with "behavior block". Does it mean that it depends on user decision? That would be weird, because I don't think WD generates such alerts, and WD's behavioral monitoring was disabled in this particular test.

Yes they used MS Edge in this test, but I'm guessing that system wide SmartScreen should still step in if people download and run malware, no matter which browser was used. But at least WD didn't perform as bad as Zemana, Webroot and Malwarebytes. I thought they were supposed to have effective protection, specifically against ransomware, but they all failed miserably.

 

WD + SS blocked 14 samples by behavior. WD w/o SS blocked 24 samples by behavior. This implies that 10 of the samples were blocked by SS.

I infer that behavior blocking in the context of the test means the malware was detected by other than signature means. This would mean rep scanning would be considered behavior detection. Additionally, heuristic scanning which WD does employ is considered behavior detection.

Most AV products including recent vers. of WD employ two types of heuristic scanning; local and cloud based. Local heuristic scanning is initially performed on an unknown process. If anomalies are detected, the file is uploaded to the vendor's servers for more detailed scanning. While this is taking place the unknown process is suspended and sandboxed(virtual not physically like Sandboxie). The vendor's server replies back to the originating device the status of the detailed scan - OK, malicious, suspicious. In most AV products, an alert will only be generated for the suspicious category along with a recommendation on what to do. The recommendation is usually to block the process. In the MRG test, an incorrect behavior recommendation is considered a failure.

Microsoft just introduced cloud heuristic scanning to WD on Win 10 ver. 1607 and it is hidden and disabled by default. As such it, it would have not been employed in the MRG test. All AV labs including MRG, products are only test at vendor default install settings. Therefore it can be concluded that the behavior detection recorded in this test was performed by WD's local based heuristic scanning.

 

Are they turning off UAC for these tests? I don't think any sample would run with that set to max.

 

It is enabled by default; "Block at First Sight" wont be enabled if the user change the "Automatic sample submission" to OFF.

More info here:
http://www.winhelponline.com/blog/defender-block-at-first-sight-cloud-protection/

 

Thanks for the clarification.

So for the MRG test, WD cloud scanning was indeed enabled.

Also of note from the link reference is that only Win 10 1607+ employs actual process suspension while cloud scanning is in process:

-EDIT- Also WD cloud time scanning has option I believe to set the duration of the scanning which is very odd indeed. In any case, it can be extended in length indicating the default setting is not the most thorough scan.

 

Webroot was removed from the MRG report so please see here for more information: https://www.wilderssecurity.com/thr...on-update-thread.364655/page-134#post-2702185 and the newer PDF report: https://www.wilderssecurity.com/thr...on-update-thread.364655/page-134#post-2702466

 

OK I see, I do know that local behavioral monitoring can be enabled in WD, but it's not enabled by default. And apparently cloud scanning was also not enabled, so I think you're right about WD's heuristic capabilities being labeled as behavior block. Perhaps in the next test both features should be enabled, I'm sure they will then score better.

Wow, this doesn't make MRG look good, but at least they admitted their mistake. How difficult can it be to setup Webroot or any other AV, it's a bit weird.

 

Personally, I don't know what this means since if their was indeed a major test setup issue, MRG would have corrected it and retested all products which obviously they did not:

Appears to me that Webroot's default config. needs to be "tweaked." AV Labs test with default "out-of-the-box" settings.

 

And: "In many cases it appeared that WSA wasn't fully active in their testing environment."

 

Emsisoft is not participating in MRG tests. What is the reason behind this?

 

Anybody know which version of Malwarebytes they use in this test ? on-demand means free version, right ?

 

Start reading from this posting: https://support.emsisoft.com/topic/15920-mrg-effitas-q22014-test/#entry123491

 

Ver. 3.1.2.1733. Assumed it was the realtime ver. since the majority of the test samples were URL based.

 

Thank you bro for sharing this. Much appreciated