MRG Effitas 360 Assessment & Certification Programme Q 1 201 7

This is just me. I don't use UAC or SmartScreen but that's my personal preference. Now that may be a good poll to setup to see how many use UAC or SmartScreen or even both or just one of them?

 

Why on earth would they test this, isn't it obvious that SS would block quite a lot, and that UAC would popup quite often? The goal of MRG and others is to test signature/heuristics/behavior blocking capabilities of AV's. They should be able to identify malware, and give a low rate of false positives. Just because SmartScreen or UAC popups, doesn't mean it's actual malware.

Good point, who know how many people have disabled it. To me they are producing useless noise. Not to speak about false positives.

 

I agree about turning it off for other av's and leaving on for the WD tests only.
BTW the new insider builds have changed some wording in IE now. Before in tools it just said turn on or off WD now it says WD smartscreen filter.

 

Its the default settings (UAC and SmartScreen ON), so this kind of setup (Windows Defender + UAC + Smart Screen + block at first sight) is what the majority of windows 10 users are running.

Wilders security members are atypical, most of us dont need "special" setups, I am more interested in pratical solutions that can be effectively used by average users.

 

I fully understand that's why I said it's my preference! Even the PC's I clean and repair and install WSA I leave UAC and SmartScreen on for the happy Clickers.

 

UAC should be disabled for all product tests. You are testing if a product can detect malware regardless of it tries to elevate privileges to run.

 

Even if average user would use 3rd party AV solution SmartScreen would be enabled (since those solutions usually don't disable it). So if it was enabled for all tests, we wouldn't know how much protection is provided by AV and how much from SmartScreen. If only WD was tested with SmartScreen, we would compare oranges to apples.

 

You don't even have to take it to that level.

"IS WINDOWS DEFENDER/SECURITY SYSTEM A THIRD PARTY ANTIVIRUS?"

No? So what's it doing in the test pool? If you are testing a population of a specific strain of fruit fly, and you allow just one fly of a different subspecies in there, your test is contaminated. Bad analogy maybe, but purists, help me out here!

I agree with those who advocate for a separate test protocol with Windows Defender/Security system only and how it would fare with "out of the box" settings, in a "real life" test environment. But, like is said repeatedly, I smell money in there, somewhere.

 

More akin to oranges and tangelos. Again, native SmartScreen is Edge's malware web filtering method. Another argument would be to disable the web filtering and reputation scanning in non-WD AV products since same are disabled in WD.

Maybe the best overall solution for the AV Labs is to stop testing WD for comparative tests since its protection model doesn't conform with the rest of the AV industry.

 

On the other hand, it's certainly understandable that some third parties may want Microsoft in there, it makes their products look gorgeous by comparison. True that. Don't apply some lofty standards and pooh this, you'd flip your lid, you would.

 

Nope.
SmartScreen is a system wide function in both WIN8.1 and WIN10.

On WIN10 it can be configured separately, for Edge and for the OS
So users of other brothers have the advantage of smart screen as well.

Turning of SmartScreen brakes the security concept of WIN10

 

Agreed. If you are testing malware you have already consented to having it run. UAC changes nothing here. UAC does not detect malware. I don't know why people think it will affect the score of anything.

 

Edge uses smartscreen.exe which is Win 10's native SmartScreen. You can disable it's use in Edge but system wise but it is still running unless disabled via Control Panel option. If native SmartScreen is disabled via Control Panel option but enabled in Edge, Edge will start up an instance of smartscreen.exe when Edge starts up unless specifically disabled in Edge itself. The same occurs for Win apps startup if SmartScreen is enabled for those.

IE11 on the other hand, uses it's own internal SmartScreen processing. Enabling or disabling Win 10 native SmartScreen processing has no effect on IE11's internal SmartScreen processing.

I personally want nothing to do with Win 10 native SmartScreen since its primary purpose is to gather telemetry data for Microsoft. I have native SmartScreen disabled, do not use Edge, and use IE11 instead. I have also disabled the native SmartScreen data upload process in Scheduled Task manager.

Some additional facts about Win 10 native SmartScreen i.e smartscreen.exe. It runs as a medium integrity level unprotected process; i.e. UAC level. As such it can be easily terminated or suspended by malware or far worse, hijacked.

 

You don't want to recommend this to user users? Don't you?

 

It also detect executables downloaded from other browsers , it is why it is system wide.

That is my point or if they want test it as stand alone product, they must compare it with other similar products.

 

I don't consider SmartScreen as web filtering of WD. If it was integrated with WD, than it would be disabled the moment you disable WD (same as with other AV solutions).

Probably it would be best (at least for Windows 10). But then I'm sure there would be questions why WD is not tested also.

 

Why? Test it in isolation, it is unique as a security solution. Unless you just want to test the detection ability of the scanner then yeah. The more I think about this, the more goofy it seems that Windows 10 security is tested in the same pool as third party.. SmartScreen is a security feature even if it's exclusive of Windows Defender, it's all under the Windows umbrella. You can uninstall Kaspersky but not Defender. I love analogies, here's another:

I want to do a maze experiment, but to my lab rat subjects I am going to add a hamster. Hey, why not, they're all rodents. Hey, why not, they're all antiviruses. There is way more to this, but can't speculate there, right? Maybe this very issue was discussed before, I can't imagine that it wasn't.

 

The fact remains it would be more relevant to test the efficacy of Security Applications on Windows Operating Systems that are set at their default settings, this is more representative of a real world test.

Charts of comparative scores for programs based on current testing methodology, which deliberately dumb down inbuilt Windows Security are interesting, but less relevant to a real world scenario.

 

i personally agree with you but most people read those tests for the detection score, so...

So do i , i think they still believe that WD is independent as MSE was...

That is obvious but guess what? not for those test labs...

Totally agree.

 

With respect, in today's ominous Internet climate, it would behoove test organizations like yours to provide results that are applicable to the majority of end-users! In the real world! Is that not your primary mission? A "heavily modified VM" is not real world although you say it mimics it. People pay attention to the graphic results and tailor their security solutions accordingly. How close are they to the "real deal?"

 

I never take any notice of these "tests" however I do enjoy reading the posts that follow them. If the AV that is the users choice scores well they're very good tests however, if the test doesn't rate that AV as the best or at the very least 2nd best then the results are questionable!

 

As soon as I saw results, I somehow new how this thread was going to develop.

 

In fact the score doesn't matter , because it is a static snapshot of the reliability of the product on a given point of time with a specific set of sample; what interest me is the methodology used and we can say that it is not thought enough.

 

Yes exactly. Again, the goal was to test AV's. Obviously, Win SmartScreen is not an AV.

Yes it's so silly. Let's face it, what if Win Defender would have had a 100% score? No-one would have even mentioned UAC and SmartScreen, you would only see comments like "no third party AV is needed anymore" and "Win Native security is so freaking good". Luckily we have MRG to expose the truth LOL.

 

Some of us care much more about the protocols, data presentation, explanations--and next to nothing about Brand X performance, that includes Defender. I do care that the security features in Windows 10 are manipulated and the corresponding ones in the third parties--are they? How large is the gap between the hype and a product's default, intact "real life" performance? We many never know, it's such a closed door to us. Virtually zero transparency yet there is influence of a lot of revenue going on.

How can anyone justify running a comparative study when the subjects consist of one hamster and a dozen lab rats? You are testing security of the operating system using a specific product, no? Not antivirus, security! With the next OS comes inbuilt mitigations. Test Windows 10 as a product, then, not just Defender. Still a hamster but the playing field is a bit more level, I think.

I edited for clarity.