MRG Effitas 360 Degree Assessment & Certification Q4 2016

OK, but in this thread we're talking about one report. And they don't mention that they disabled SmartScreen in that report.
Also if Smartscreen prevents proper testing it's normal to disable it. They are not testing how secure system is but how much security specific AV / AM provides.
I agree that Smartscreen would improve "detection" of all AVs but it would hide information how much security specific AV/AM brings to the table.

 

All three MRG reports are mentioned and discussed in thread before I entered this thread, so naturally I look at all three reports.
Smartscreen doesn't "prevents proper testing". SmartScreen handles URL-filtering, blocking of unknowns, low rep or malicious executables and a lot more.
That's a lot to disable and then claim that a test is "real world".
WD and SmartScreen both do well on their own and there are some overlap between them, but they are meant to complement each other.
It's really not difficult to add a column to each test where all native security are enabled and allowed to complement each other, or if only one column are wanted then stop disabling modules and test Windows security as a whole.
Considering how all testing institutions want their tests to be "real world", then it really should be within their reach to do this.

 

Obviously Smartscreen is disabled. if not, most samples won't even be able to be executed.

the methodology isn't very clear : type of account, admin, SUA ? what UAC level? Smartscreen , yes, no?

About smartscreen: to be effective the samples must be downloaded from the net to the test machines , not imported from an external machine. If it is , samples won't have the "mark of the web" , hence won't be flagged by Smartscreen.

then after knowing that :

so obviously , Smartscreen wouldn't detect them if enabled.

Then we have 2 opinions:

1- those thinking Win Def is a standalone solution independent from the other native security feature of Win10, so it has to be tested alone. ... ummmmm but this is WinDef (win8/10) not MSE (Win7)...
2- those like me, saying Win Def is part - complementing and is complemented - of the native security of Win10 (which is obvious).

Logically, all the native features of the OS must be enabled during any test; because if you install Win10, WInDef, Smartscreen and UAC are enabled by default. If one of them are disabled, test is flawed.

To be a valid test, the samples should be able to bypass SS and UAC, then tested against the security solutions (including WinDef); then you can say which one bypass the protected system or not. Then you can have a realistic impact of the product on the system's whole security.
Guess what? it would be difficult to find a big amount of samples able to do this...

Now if the labs just want to test one component of the security (say scanner) , it must be done for all products. You want just test Win Def, so just only use cloud & scanner of any other products, not the BB/HIPS/webfilter or whatever.

All those tests labs are not better than Youtesters , they just have more machines...

 

I agree.

 

In regards to the MRG banking tests, I note the following.

MRG was the first AV Lab to create such testing. They created their own methodology to do so. Over time, they have enhanced it adding botnet and a few "synthetic" malware tests. One does not have to accept any or all testing methods employed. I for one find issue with the use of "synthetic" malware.

However, there is one indisputable fact. All tests are employed equally against all comparative products. For someone to state that they adversely affected one product and not likewise the others is well .................... ludicrous.

 

@guest :
I fully agree with you.

Current way of testing from the testing institutions has zero value.
They test vendors with all modules active against a single module of the OS.
It's like testing 20 cars for max speed and maneuverability, and then they remove two wheels on one car but not on the rest.

Anyone with hands-on experience, can see this kind of testing is flawed.
And from a comparative point of view, absolutely nothing useful can be concluded in current state.

 

I will also advise individuals making blanket statements such as "SmartScreen" was disabled and the like without any such documented proof of same are libelous.

 

I agree it's not real-world situation for testing system security but it's more realistic when testing AV only. If SmartScreen was enabled then it would have to be for all AVs (it's "real world" situation and 3rd party AVs don't disable it). If they decided to use SmartScreen then ALL AVs would perform better since they would benefit from SmartScreen blocked files. And at the end AVs with better detection would still perform better than WD but difference would be smaller (only my logical speculation). Also should all notifications for legit files from SmartScreen be considered as false positives?
WD uses SmartScreen to enhace system protection but SmartScreen is not part of WD. Since they are testing protection provided by AV/AM and not system protection it's IMO logical to disable it.

 

Value of test depends on people that read it and what conclusions they can make from results. I know it doesn't provide any answers to your questions, but it can give some info to rest of us.

 

Perhaps @Sveta MRG could chime in and answer some of the burning questions being raised in this thread.

 

All native security complements each other in actual real world use.
What is the point in comparing one OS module against vendors with all modules active ??

As you can see in every test report thread :
- It just confuses end users.
- It has no comparative value since test does not reflect daily use in the real world that we all live in.
- The only users that are jumping up and down each time these flawed reports are released, are the usual 3-4 anti-Microsoft users that haunt every security forum hoping someone will start a fight with them.

So I still wonder, what is the value in testing a OS in a semi-disabled state that absolutely no end user will ever use it in ??

 

Frankly I think it would be more exciting to hear comments from the vendors being tested.
What is their view on current status ?

 

Value of test is relative to each user. Personally I use Windows 7 and don't use IE (as most people for whose systems I take care of) so this test gives me more useful data than if SmartScreen was enabled...

 

read carefully :

Where do you see any mention of Smartscreen? it should be at point "a" , they use Edge guys ! not IE or Chrome or FF ; Edge !
Do i remind that Edge is supposed to use Smartscreen and if Smartscreen is enabled it should flag the file ? so either they disable it , or either they clicked "yes" to allow the sample, so test flawed by design.

Now if i'm wrong , i invite the test lab to correct me.

It doesn't mean that the vendors results are bad, it means that WD which depend of the whole native security got an handicap from the start. If it was MSE , i won't even oppose it, because MSE isn't a native security feature.

 

Hi All,

Both UAC and SmartScreen are disabled, if they were kept on we wouldn’t be able to measure the real performance of an AV/ISS product as UAC and SmartScreen would interfere.

We are testing performance of AV/ISS products and not Operating System components.


Cheers,
Sveta

 

Windows Defender is now an OS component on Win10 so you can't compare it anymore to others. However, testing MSE on Win7 will be legit.

 

In regards to whether Win 10 native SmartScreen processing was disabled or not, it is immaterial as far as these recent MRG tests are concerned. Here's why.

Win 10's native SmartScreen protection performs a reputation scan of a process at startup time. This by definition means the file had to be downloaded in the first place. This is all Win 10's native SmartScreen processing does.

The SmartScreen processing built into both IE11 and Edge is separate and distinct processing from Win 10's native SmartScreen processing and is built into each browser. It is enabled by default in Edge and also in IE11 when it is configured to use recommended settings which I assume was the case for this testing. The browser based SmartScreen protections include anti-phishing, exploit, drive-by download, plus web based malware and malvertising protection. Additionally and most important, the browser based SmartScreen protection performs the same reputational scanning on file downloads as that done by Win 10's native SmartScreen protection.

Bottom line - with Win 10's native SmartScreen protection disabled, it would have no impact on these MRG tests since they were using IE11 and Edge for this testing. So it's time to stop this nonsense.

Additionally, I have used browser based SmartScreen in IE since XP days. I also use Edge. I always used it enabled and with other AV software that employed web filtering protection. Never once have I ever received an alert from browser based SmartScreen other that a "once in a blue moon" alert about an unknown file download.

 

Exactly my point, problem is that results are given in % ; and in the user mind , anything less than 90% suxx.
Now see the impact it has on WD, people will believe it suxx , then will use a 3rd party vendors , and spread the myth that WD suxx. Who benefit from it ?

 

That is Smartscreen goal , if they are legit files why would you get an alert?

 

Official explanation in post #40, that UAC and SmartScreen was completely disabled during testing.

So no reason to call anything nonsense.

 

It was so obvious that they were disabled...i am still amazed that people didn't saw it.

 

The difference between AV rep scanning and Smartscreen is many AV's will perform behavior analysis on an unknown process to detect suspicious activity. The AV's will then perform further reputation analysis as to widespread use of the software. It then uses both factors to determine is the software is safe for use. The final determination will be to allow the software to execute or to alert the user, display the analysis result, and let the user decide to allow or deny execution. This processing greatly reduces the occurrences of false positives.

 

I agree, @guest.
Luckily we now have an official statement on page two and Wilders don't have to have one of those 5-10-20 page long threads with heated arguments back and forth.

 

Clarification. Did you disable SmartScreen in both IE and Edge(don't know how that is possible) or only Win 10's native SmartScreen protection?

 

And since they "interfere" with the test , this is a official proof of their efficiency.