MRG Effitas 360 Assessment & Certification Programme Q2 2017

https://www.mrg-effitas.com/wp-content/uploads/2017/08/MRG-Effitas-360-Assessment_2017_Q2_wm.pdf

The big new is ................ Windows Defender was tested with SmartScreen enabled and as expected, it scores in the "bottom of the heap." Kudos to @Zoltan_MRG to listening to Wilders comments in this regard.

At least the test shows that SmartScreen blacklisting is moderately effective in a 24 hour detection time frame except for ransomware.

 

Kudos to MRG for listening, as expected Smart Screen made a massive difference in detection/protection (except for PUP), but it wasnt enough to beat Kaspersky and other top solutions (as expected too).

 

Thank you @itman for posting.

 

Yes, thanks, itman, very interesting.

 

Interesting (and somehow expected) results, thnx

 

at least with WinDef + SS , you have now a free decent protection out of the box, those scoring lower dont deserve a cent and should become free products

 

One thing that should be noted that this test graphically illustrates is the following.

This test was done on Win 10 with Edge as the browser employed. So both native and browser based SmartScreen were being employed in regards to Windows Defender detection scoring. If you are using a non-Microsoft browser or SmartScreen is disabled in IE11 or Edge, the Windows Defender column w/o SmartScreen should be used in determining what protection is given.

 

What the hell, haven't read the report yet, but I'm a bit shocked at how badly SmartScreen performed, why did it still fail to block 19 samples? Then you're definitely better of using a third party AV. Or perhaps WD fanboys can come up with an explanation.

https://www.howtogeek.com/123938/htg-explains-how-the-smartscreen-filter-works-in-windows-8/

 

Kaspersky the only level 1 hehe

 

SS work best if the malwares are downloaded from the test machine, not imported from network or external storage. See "mark of the web."

Also 24h is maybe too short. Remember that MS focuses on prevalent malwares, not zero days.

 

I think you're stretching a bit here. The MRG report states that 10% of the samples or approx. 36 were delivered via USB. However, all the samples were originally downloaded via URL and then placed in an archive. As such, they would still have the "mark of the web" associated with them.

 

which packer does keep the "mark of the web"?
WinRar and 7-zip do not...

 

The one integrated on Windows or this one https://www.ponsoftware.com/en/

 

You're correct. It's a great "mark of the web" stripping technique and as such applicable for malware testing since no other security product is conditioned by it.

 

Just tested it on Win 10. When the file is extracted, it no longer has the "mark of the web" associated with it.

 

On Windows 10 ?

 

basic methodology is that you open urls and download samples on the machine you test the apps, you dont import them from another, especially if the tested apps is relying on the mark of the web. Unless you want test the detection towards external drives.

things must be done properly and transparently.

 

Smartscreen was a huge improvement for Windows Defender. And if it's true that there was some issue with the mark of the web, then the result are even more impressive.

 

Kaspersky - no surprise, Avast benefits from AVG merge (and AVG as well).

 

Where in the MRG report do you see this? Page 6 of the report states that all USB samples were archives originally downloaded via URL. That implies they were done so on the test rig. The archives were then copied to a USB drive which by the way does remove the "Mark of the Web" from them. It is assumed that then the files from the archive were extracted and run.

If WD was doing its job properly, it should have detected the malware in the archive when it was created via download from the URL. If it did not, WD should have detected the malware when it was executed from the USB drive.

The most important point however is this. The MRG report does not state by device category what malware was or was not detected for a tested security product. So it may very well be that WD did detect all USB based malware.

-EDIT- Additionally, as noted in my reply #12, the total number of USB archived samples were 36. WD + SS total missed detection number for the entire 360 test was 19. Therefore, it is fairly obvious that WD + SS detected some if not all of the USB based malware samples.

 

So a new MRG test is out.
Good to see that MRG are now getting closer to testing the combined Windows 10 native security.
Let's have a look at what the report actually say.

First of all, MRG Q2 2017 test report are done using Windows 10 1607 Anniversary Update.
Technically Windows 10 1703 Creators Update was already available in late Q1 2017, but MRG had to setup test environment and perhaps there was not enough time to get everything ready. So fair enough.
The MRG Q3 2017 test must be on Windows 10 1703 Creators Update then.

Why is this important ? Well, Windows 10 1607 Anniversary Update was when Windows Defender had the Block at First Sight introduced.
In Anniversary Update, the Block at First Sight was implemented so Windows Defender would analyze unknowns locally, if inconclusive then query the cloud, if unknown in cloud, then upload the still locally blocked unknown to cloud, then release.
Minutes later the verdict from cloud would protect all other users.
This brought down response time on unknowns from hours to minutes.
But - in this first implementation of Block at First Sight, Patient Zero was not protected.
So response down from hours to minutes in ecosystem, but with Patient Zero being sacrificed.

Now move on to Windows 10 1703 Creators Update. Here Windows Defender has been improved to even better analyze locally and cloud has been updated to detonate the uploaded unknowns and run them through the full set algorithms, behavior models and heuristics in just seconds while Windows Defender still holds the unknown locked locally, then gets verdict back and then remove the still locked unknown if malicious.
Response time down from minutes to seconds AND Patient Zero are now protected.
More here : https://blogs.technet.microsoft.com...me-defense-against-never-before-seen-malware/

(And Windows 10 Fall Creators Update brings even more goodness. That branch are smoking hot).

Next there's the PUA/PUP portion of the test.
Here it's important to remember that Windows Defender will not prompt and ask if full PUA/PUP detection should be enabled and it's therefore not enabled in this test according to MRG test criteria. A user has to make that decision themselves and actively enable it manually.
Windows Defender still does pretty good in the PUA/PUP section of the test when on default settings.
But if a user want to enjoy the full PUA/PUP detection capabilities in Windows Defender, then they need to enable it manually.

Finally SmartScreen where some ask why didn't it block more ?
As can be seen in the test, SmartScreen did really good and blocked lots of malicious connections and files.
Additionally SmartScreen also handles reputation blocking.
Now remember the MRG test criteria - a block with a clear prompt saying something malicious was blocked are accepted by MRG while blocks accompanied by a recommendation are not accepted by MRG.
SmartScreen does the majority of blocks with a clear warning and are accepted by MRG.
But SmartScreen blocks of unknowns/low reputation are accompanied with a recommendation and are disregarded by MRG according to test criteria.

The solution to that for a end user are of course to never overrule a block of unknowns/low reputation - nobody wants to be the guinea pig on a unknown. And when on 1703 Creators Update or newer, then simply set SmartScreen to Block instead of Ask.

To sum up, don't stay on old branches.
- Update to Creators Update (and newer when available) and have Block at First Sight protect Patient Zero.
- Activate PUA/PUP detection.
- Set SmartScreen to Block instead of Ask.

And life is still good.

 

Nice analysis

 

Thank you, Imuade.

 

Nicely done!

It seems that you have just ruined the weekend for the MS haters