AV-Comparatives: Real-World Protection Test February-June 2018

I would also like Cylance to be tested by AV-Comp, but it would likely have to be a special-design test because of Cylance's unique methodology. However, any AV must pay to be included in tests by major labs like AV-Comp.

Does this buy-in cause bias in AV-Comp's results? I think not! Why? Partly because of certain deep pocket AVs that have dropped out of AV-Comp's tests because they didn't look good in AV-Comp's results. I have a lot of admiration for K7 AV because they know they will not be among the best in AV-Comp's tests, yet they continue to pay to participate. I think that staying in the AV-Comp "race" will, over time, make K7 a powerhouse AV. Much as I hate saying it, I believe the same holds true for Microsoft's continued participation in those tests. M$ got guts!

As far as I know, AV-Comp (user name IBK) is the only test organization guru that regularly attends Wilders & "stands up to the flak" when folks take shots at test methodology & honesty. I admire IBK for that.

 

Cylance Protect was tested here: https://www.av-comparatives.org/tests/advanced-endpoint-protection-test/

 

Looks like Cylance did pretty good to me. You?

 

Well, it was the Protect product vs. the Home vers.

For example and as AV-C pointed out, Cylance Protect scored 24/25 on the Fileless Malware/Exploit test because its default setting was to block all script execution. So if that concept appeals to you, you need to check if a corresponding setting exists in the Home ver.. Also note that the Home ver. is not configurable option-wise based on postings given in the Cylance Home thread.

 

Tested Products
All products were purchased from distributors, except for the Bitdefender product, which was provided by the vendor itself. The following up-to-date licensed product versions were tested:

• Bitdefender Endpoint Security Elite 6.2
• Carbon Black Cb DEFENSE 3.0
• CrowdStrike FalconHost 3.7
• Cylance CylancePROTECT 2.0
• Kaspersky Endpoint Security for Business 10.3
• McAfee Endpoint Security 10.5.2
• SentinelOne Endpoint Protection 1.8.4
• Sophos Central Endpoint Advanced Protection and Intercept X 11.5.9-3.6.10
• Symantec Endpoint Protection Standard 14.0

 

The thing is, when SmartScreen shows an alert about "unrecognized app" it basically doesn't know if it's malware or not. So this should be marked as a fail unless Win Defender never wrongly flags files as malware, because then you can simply block all never before seen files on first sight.

Not bad at all indeed, and there is no reason to believe that the home user version performs worse because they use the same technology.

 

@Rasheed187 , here's Microsoft's write up on how WD's cloud protection works: https://cloudblogs.microsoft.com/mi...me-defense-against-never-before-seen-malware/ . As is true with all their propaganda based publications, it highlights how the process was detected as malicious within the 10 sec. default cloud scan period. Of course, MS won't state what happens if the 10 sec. scan period expires. This is when a verdict will be rendered requiring user interaction - allow or block. Also note that the maximum scan period that can be configured is 60 secs.. So any malware that is timer execution based can defeat it.

-EDIT- Additionally based on this comment from the Microsoft article:

I am going concur with your statement that that user decision only applies to SmartScreen alerts for unknown process detection. If WD cloud scanning "times-out" prior to a malicious verdict, the malware will run unabated. One possible exception is if PUA detection is enabled and by default, WD prompts for user decision on its execution. Such prompting is the norm with most other AV products if PUA detection is enabled.

 

This seems to be the most logical to me. So if Win Def says file is clean, then SmartScreen will stay quite unless it has never seen the file before. If you choose to allow, then it it's game over because Win Def didn't find anything suspicious.

If it worked differently, then what's the point of mentioning "user depended" because Win Def would have still nailed the malware if you clicked on allow. But this doesn't seem to be the case, so that's why I say Win Def basically failed to block these 31 samples.

 

Excluding browser based SmartScreen(Edge and IE11), Win 10 native SmartScreen will scan via black/whitelist any file execution downloaded from the Internet. So SmartScreen will always scan prior to WD. You will constantly get an "Unknown" popup from native SmartScreen if the .exe is not white/black listed. You then must manual override SmartScreen to run the program. I have a test malware I use that is so obscure, SmartScreen pops up "Unknown" each time I run it.

Part of the "confusion" with Win 10 native SmartScreen is that if you are using WD, you will never see an "unknown" pop up from it I believe. Unknowns are auto passed to WD for cloud scanning. You will only see an "unknown" pop up if you are using a third party AV.

However and to add to the WD setting "confusion," there is a block at first sight non-default option of "high" that will alter WD's behavior upon completion of cloud scanning:

https://docs.microsoft.com/en-us/wi...ock-at-first-sight-windows-defender-antivirus

It appears to me that both AV-C and AV-Test are testing WD with the option set to "high." I believe this is the only way the user will get a decision prompt from WD after cloud scanning times out.

 

F-Secure is a FP monster, won't recommend it any more.

 

I have a base of over 100 F-Secure installations.
No reports of FP from any private user.

Most likely the FP occur on exotic samples...

 

it appears when browsing your daily news isn't the only thing you do every day.

 

This all is getting confusing, so you're saying that Win Def will show you some kind of alert? I assumed it was the SmartScreen alert when they talk about "user depended". Then how does this alert look like, because it really makes no sense. Most AV's will simply say whether some app is goodware or badware, they don't present any alerts about "this may be malware or not", and make the user gamble.

 

Yes and noted in the Microsoft article quote I posted. I could not find an example though of what exactly will be displayed. My guess that it will be something along the lines that WD has high confidence that the file is malware with the default action being to block it.

I do know that in WD ATP, the default confidence level is 80%. Above that level and WD APT considers the file "safe." Bumping the confidence level to "high" in WD ATP raises the level to 90%. I assume the same confidence percentages apply to regular WD. Note that with a confidence level of 90%, false positive rates increase significantly as also malware detection rates.

 

But where do you get this info from? Because I couldn't find any information about this alert, it currently seems that you're purely speculating. It still makes more sense to me that it's the SmartScreen alert, but even if it isn't, it should still be counted as a fail because an AV should never leave the end decision up to the user.

 

Is this the alert you were seeking?



It's from this document, and I *think* it's an example of what ATP produces.

Transparency report
Examining the AV-TEST March-April 2018 results
https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE2ouJA

 

No, that's an alert from Win Def ATP, not from Win Def AV.

 

Yes and no.

Here's what I have observed when running a periodic scan using WD when a third party AV is the primary on Win 10 1803. If WD sees something suspicious, it will pause the scan and connect to the cloud using native SmartScreen. From this one could infer that SmartScreen is doing the cloud behavior scanning. I say no because it has no mechanism to do so since it only uses white and black listing. What I believe is going on is:

1. Microsoft is gathering sample data for further off-line cloud scanning analysis.
2. The submitted file is being compared against the SmartScreen cloud based white/black lists with are constantly updated by Microsoft.

When native SmartScreen is deployed with WD realtime scanning, it now has the capability to interface with WD's GUI elements. As such, WD can throw an alert at completion of SmartScreen and WD cloud analysis. It could be very well that instead of throwing the SmartScreen "unknown" alert, it changes it to a WD suspicious alert. On the other hand, there is Microsoft documentation that when block-at-first-sight is set to "high" mode and a determination cannot be made prior to scan time-out period, the file will be auto blocked. This implies that one will never see a "suspicious" alert generated from WD.

Again someone using WD in realtime mode needs to show what this alert looks like

 

In regards to "mark of the web" bypasses, I found this interesting POC:

https://enigma0x3.net/

 

here something found on AMSI: Antimalware Scan Interface (AMSI) - YouTube https://www.youtube.com/watch?v=wBK1fTg6xuU

 

so whats the difference between AMSI and ASR? Attack Surface Reduction (ASR) blocks Office, scripts, and email-based threats, The Microsoft’s “Antimalware Scan Interface” (AMSI) detects malicious Powershell code (fileless or executed from drive) with functions are AmsiScanBuffer and AmsiScanString: memory scanning, URL and IP analysis, file inspection, VBA and scripting runtime analysis.

isn't this overlap of script blocking features? ASR seams to be only for enterprise (I am reading the Microsoft leaflets) and amsi is for all O/S versions which adds to the confusion

 

ASR is a set of individual rules that can be enabled one by one, and this can be done on most versions of Windows 10 by means of powershell scripts, or by using Andy Ful's ConfigureDefender tool. Read the description of each ASR rule and you will see what it does, and in most cases I think you will see how it differs from AMSI.

 

ah yes that ones, yes I wrote some rules on it (I will post it later), I forgot its ASR (my memory), thanks, so AMSI is in all versions or APT, should be in all, I believe AMSI is not complete on non enterprise version (win10) and complete on enterprise with added ip and url scanning (Machine Learning) right, if they send files to cloud in all versions of AMSI, the only difference is the url/ip scan thingy ?

 

AMSI; i.e. Anti-malware Scan Interface.

It is a kernel memory sandbox that allows WD and other AV solutions to examine packed, encrypted, and obfuscated scripts prior to actual execution after they have been unpacked and unencrypted. It only examines scripts run by the following script engines; PowerShell, wscript, and cscript. For example, it does not intercept Python scripts.

AV solutions continue to have problems with highly obfuscated PowerShell scripts. Obfuscation refers to the ability to randomly place select characters within a PowerShell script to obscure the actual script code. PowerShell ignores these characters when it parses the script. Microsoft's solution to this issue was to create an ASR available in 1809+ Win 10 versions to block execution of all obfuscated PowerShell scripts. A bit "lame" but nonetheless effective since obfuscated non-malware PowerShell scripts are a rare occurrence.

BTW - ASR mitigations are only applicable if WD is used as the realtime AV solution. You can create them w/o issue but are only referenced by WD.

 

thanks , as per usual your comments are of great value
some bypassing techniques I've found:
https://resources.infosecinstitute.com/antivirus-evasion-tools/