MRG Effitas 360 Assessment & Certification Programme Q 1 201 7

Hi All!

First of all, we would like to thank to all of you for pointing out mistakes in the report. Financial malware chart: Webroot and Microsoft fixed. Some behind the scenes story: Excel can't generate automatically graphs we need (I am talking about mostly design), and sometimes manual Photoshopping is needed, e.g. to insert the table into the image. This is manual work, so we made a mistake. Fixed now, thanks for letting us know.

Regarding the two test scenario mentioned:
1 - Windows Defender standalone (already tested)
2 - Windows Defender + Block at first sight (default settings) + UAC + SmartScreen - "special scenario"

We also believe this could makes sense, so we might introduce it in future tests. This is still a discussion. We already discussed this 1 year ago and voted for a no, but maybe this time.

Regarding comments like:

This is a pretty vague accusation. I have three main points here:

1. When it comes to other testers on Youtube or Unbelievable demos at conferences or other test labs: Whenever it is mentioned that test was done on the following hardware spec .... you can't be sure whether the test itself was on physical machine or virtual machine. Unless it is stated explicitly that test was done on physical machine, 99% of the tests are done on VMs. And I am not sure how many testers harden their VMs, but what I have seen so far, these are default VMs with Vmware tools or Virtualbox guest additions installed. At least we are honest about it by writing every detail we think is important - but all we get is bashing. Some people are too emotional and less objective whenever it comes to us.

2. If you run Pafish on our test VMs, you get only one warning regarding VM is detected. This technique is used in <1% of malware. All the main VM detections are avoided in our tests. Also please note that this issue can only change the results of behaviour protection, because malware does not execute correctly, thus behaviour protection can't kick in. For future reference, please check the amazing https://github.com/hfiref0x/VBoxHardenedLoader project. This is what we use.

3. VM detection is the past. We use multiple techniques to make the test system look like a real desktop machine with user activity, and meanwhile we make best efforts to hide all the test tools and monitoring tools we use for the tests. I do believe no other test lab does this. If you are a test lab, and you do this, please comment, I would be happy to know this. If you are interested in technical details, please check my project here: https://github.com/MRGEffitas/Sandbox_tester
We do everything possible to avoid being detected as a test machine. And guess what, even physical machines can be detected this way. Needless to say, AMTSO recommendations are outdated.

To answer the "A "heavily modified VM" is not real world" comment, I would like to rather know how you propose to test, rather than saying this is not real world. Without making detailing any comments or facts on how our setup changes the results, I can only ignore comments like this.

 

Nice! This is / could be good news.

 

I'm totally ok with that

 

Hi Zoltan_MRG: Thank you for your response, which was more detailed and informative than I expected. I appreciate that. Vague, yes, because there really isn't a lot of solid, detailed information about test methods and behind the scenes stuff available to us. I am not an "insider" in this matter. I rely on what you tell us. Having conducted tests in other contexts, this was of abiding interest to me. If my persistent comments annoyed or disturbed you, I apologize. But I hope you can appreciate the interest, it is better than apathy by far.

Thank you again.

 

Thanks Zoltan_MRG, it is very much appreciated that you are considering this new "scenario" test for Windows Defender.

 

I've just read about the "Block at first sight" feature which is basically cloud detection. Why did you disable this? I believe all components of AV's should be used. But UAC and SmartScreen are no components of Win Defender. UAC doesn't even identifies malware, so it's not relevant. But it would be nice to test Win SmartScreen, it's basically a white-list from what I understood.

 

Re: virtual environment: I had a dedicated cheapo a couple of years ago, and from that, developed a healthy respect for malware. I suppose I could put VMware back on here but it would be pointless as I'd never test malware using this specific machine, virtual or not, and I have no other use for it at this time.. It's something, isn't it? Still remember the speed and efficiency of the infections and this was two years ago. Sorry again for being so persistent, but I'm very pleased MRG listened.