MRG Effitas 360 Assessment & Certification Programme Q 1 201 7

People can have different opinions regarding the whole SmartScreen and UAC debacle.

However...

If this is true, then these test don't accurately reflect WD's effectiveness at all. They might as well disable the proactive components of all the other competitors (for example Bitdefender's Active Threat Control) if they want to be a little fair.

Though it would be better if they actually test WD with Block at first sight enabled.

 

There still is one thing unexplained in this test.

How can HMP which is an on demand scanner be 100% against ransomware tested? Is the test for the detection of the ransomware itself regardless of the fact that all your files may have been previously encrypted prior to detection of the ransomware?

 

Yeah, itman, I asked YOU that on page 1 already, lol. Page 10 in the pdf, right? Read the methodology page?--yep. This whole thing baffles me-- time to blow it off in its entirety. STATUS QUO, unless you're into all them pretty graphs. Then, by all means, pick 'n' buy.

 

Only thing I can assume is MRG downloaded the ransomware but didn't execute it until a reboot was performed. HMP default scan option is a boot time.

-EDIT- AV Labs use payload ransomware samples since its impossible to acquire the actually dropper since its usually part of an e-mail. Origin sources never release the actual e-mail due to privacy reasons. At best, they might have the e-mail zip attachments, etc.. I assume the MRG testing involved loading the ransomware samples on the PC. It was then tested if the sample was detected upon creation by signature.The test rig was rebooted. They probably then ran an on-demand scan for each AV product tested. If the ransomware wasn't detected by signature at that time, the ransomware sample was manually executed to see if it was then detected.

In reality the tests don't reflect how real ransomware is delivered or executed for that matter since the dropper within an e-mail will download the payload with a means to execute it prior to any reboot in a number of ransomware variants.

 

I just loaded mine and here are the default settings "out of the box".


Very baffling how they conducted and interpreted the tests. As you said, just the payload is likely tested, which is a fraction of the process. Does anyone have just an on-demand scanner on the system and that's it? I don't see how this can translate to test validity.

Edit: this is the interface when you've checked scheduled scans yourself. Now, the boot scan is enabled.

 

It reminds me of a test a while back @ Dennis Labs which showed the free version of MBAM 2 had better protection than either MSE or WD, I can't remember which now. That was a weird result!

 

They run there test in a VM, that disqualifies the"test" anyway

 

Well, the questions were specific to the ransomware portion of the test, so a virtual test environment shouldn't be a disqualifier of itself, right? Overall this is too fuzzy anyway, blow-off time, unless anyone has the magical definitive explanations for the ransomware findings.

 

+1

VMs aren't real systems, that is it.
MRG effitas = youtesters methods

 

guest, you once said that in the past Comodo didn't worked properly in VMs, right? Are you aware of any other security product that doesn't work properly in VMs?

 

@Azure Phoenix Some people observed that Webroot cloud has issues on VMs

 

No AV Lab runs tests in a VM. It is against AMTSO guidelines; MRG is an AMTSO member.

The standard AV Lab procedure for testing is:

1. Dedicated stand-alone hardware.
2. Fresh install of the particular OS ver. being tested.
3. Fresh install of the particular AV product being tested.
4. AV product tested with default settings.

Also, cleaning procedures are implemented after each AV product test such as disk wiping or replacement to ensure no residual malware traces remain before the next AV product. In most AV labs, enough dedicated hardware exists so that thorough cleaning only has to done after all comparative products have been tested.

 

yes sure....

https://www.mrg-effitas.com/wp-content/uploads/2017/05/MRG-Effitas-360-Assessment-2017-Q1_wm.pdf

first line of the methodology.

 

Okay. Looking at a previous MRG quarterly pdf, virtual machines were used then.



Surely AMTSO would have sanctioned this by now, no? Admittedly, I'm in unfamiliar territory, so please understand the interest.

 

As far as AMTSO dynamic testing guidelines, scroll down to the Virtual Machine section of this document: [URL]http://www.amtso.org/download/amtso-best-practices-for-dynamic-testing/?wpdmdl=1149[/URL] . Although they don't specifically prohibit VM use, they also do not recommend it.

Perhaps @cruelsister will chime in. She tests ransomware in a VM and based on her web postings, she has had zip success with getting VM aware ransomware payloads to run.

 

People reading those test:
- shouldn't take those tests as truth but just as indicator of the reliability of a product in a given point in time with a specific set of sample; nothing more , nothing less.
- must be aware that those tests promote vendors (indirectly or not).
- must be aware that no indications of the used samples are available, nor their sources. From this, the reliability of of the test can't be verified.
- must be aware that essential security features of the OS are disabled (sometimes without being mentioned), giving an handicap for a product built-in the tested OS (which is relying on those feature).
Since those features will works whatever a 3rd party product is present or not, the use of the said features won't handicap the 3rd party ones.
Based on this, disabling them clearly indicate that the testing entity is biased.
- Since cloud features are used , if a product share its cloud , there is no indications that other products using the same cloud won't benefit from it.
- must be aware that the test results of some labs are sent to vendors before public publishing, hence a vendor can request a new test (which may be paid)

 

Oh wow, that group never occurred to me, that technically should be included if you want a fair sample array. OK. Never mind about the virtual machine angle.

 

As far as Win 10 native SmartScreen goes, my opinion is it should always be enabled for AV Lab tests based on the following:

1. It is a build-in OS protection mechanism. It is enabled by default on all Win 10 installations.
2. In reality, it will probably have zip effect in interfering with other AV vendor software detection. In every AV product I have used on Win 10, its protection mechanisms detected prior to any alert by native SmartScreen. In fact, the only way I could determine if native SmartScreen actually worked was to disable the AV web filter, rep scanning, etc. at which time I did receive an alert from SS on a test malware download from the browser. As far as its exploit, drive-by download, etc. claimed protections, I have never received an alert other than by use of the Microsoft test page for same. Note: to date I have never seen an independent test done as to native SmartSreen effectiveness for its claimed protections.

 

BTW - it's not just ransomware that can be VM aware. Banking Trojans such as Dridex are both VM and sandbox aware. Ditto for almost all APTs.

 

OK, I will try to explain this in a few sentences.



First of all, I’ve said this before and I will say it again. We are testing the efficacy of Security Applications and not OS components. UAC and SmartScreen would heavily interfere with the results and one could not interpret the results corectly. So again, we are testing Security Applications without the assistance of OS components such as UAC and SmartScreen.

I have personally spoken to about 20 security experts regarding this and they all agree that in these tests, using UAC and SmartScreen would change the results and won’t represent the true capabilities of a Security Application tested.


Virtual Machines, you have to understand that our systems are MUCH MORE COMLEX than you think, in theory we do use VM’s but they are HEAVILY modified so that VM aware malware can’t detect it’s a Virtual Environment and runs on them like it’s a physical machine.


We added more details into the results to show various layers of protection in action, don’t think that if something is not green that it’s a fail.


Cheers,
Sveta

 

I have a feeling it wouldn't have helped Win Defender. VM or not, it would have failed. I mean other AV's didn't seem to have any problems detecting most of the samples?

 

I buy into the UAC argument.

What has not been fully explained is why can't native/Edge SmartScreen be enabled just for Windows Defender testing? If so, it would certainly end these protracted WD results discussions for each one of your tests. This mode is also how the average user would be running if they were using WD as their AV solution.

The problem is Edge uses native SmartScreen as it web filter and rep scanner. The same is true for the IE11 and WD combo with the difference that SmartScreen is built into IE11. The argument being made is since Microsoft does things differently from the way other AV vendors protect, they are being penalized for doing so. Doesn't make a lot of sense to me.

 

Sveta, I totally agree. There are certain people who can't accept the fact that Win Defender does a bad job, it's very sad. I tried to explain to them that SmartScreen is a separate component that also works with third party AV's. And you know what the funny thing is, if WD would have nailed this test, nobody would have even talked about the disabling of UAC and SmartScreen LOL.

 

Without UAC and SmartScreen those tests dont represent the DEFAULT and FREE protection built-in modern Windows SO (W10 Creators Update).

Why not make a special test scenario?

1 - Windows Defender standalone (already tested)
2 - Windows Defender + Block at first sight (default settings) + UAC + SmartScreen - "special scenario"

I really want to see this scenario tested, while it doesnt reflect the capability of Windows Defender, it will represent the true capabilities of Windows 10 security.