Thanks for posting this. Not digging the fp's by CrowdStrike. ESET and MS looking pretty good though.
Interesting test, but it's not clear to me what the difference is between Real World Protection and Malware Protection? It almost seems like the first is about the ability to block malicious URL's, who cares about this? And when it comes to malware protection they used 1007 samples, so why not report how many of them were missed?
Thanks mini. Boy, business AVs sure is a crowded field! BTW - I'm pleased that Avast did well (I use AVG, Avast's twin sister).
https://www.av-comparatives.org/real-world-protection-test-methodology/ https://www.av-comparatives.org/tests/malware-protection-test-march-2018/ In summary,
GDATA. Yes. "If a product does not prevent or reverse all the changes made by a particular malware sample within a given time period, that test case is considered to be a miss" What's that "given time period", 1min? 5min? Reverse all the changes - what's "all the changes" in here? Registry, appdata etc locations? There's always be a remnants of the non-active malware(s), how this is measured?
If it's a cloud based detection aka BB offline test. On my own malware test(s), GData does an online (cloud lookup), then it does an offline behavior blocks with its Deepray. Worked really on my own offline BB testing procedure. Its not perfect tho, i mean GData BB, when i ran magber ransomware, which uses simple vssadmin to encrypt files, files where encrypted. Oh no. 12 hours after, the BEAST killed it.
Thanks, it's now more clear and I guess I was right, the first test is more about blocking malicious URL's, but browsers can als do this. I'm only interested in the ability to block malware from running once they get downloaded on the system by user or exploit. So in my view the Real World Protection test isn't needed.
from the introduction which can be found in the real-world protection test reports: "In this test, all protection features of the product can be used to prevent infection – not just signatures or heuristic file scanning. A suite can step in at any stage of the process – accessing the URL, downloading the file, formation of the file on the local hard drive, file access and file execution – to protect the PC. This means that the test achieves the most realistic way of determining how well the security product protects the PC. Because all of a suite’s components can be used to protect the PC, it is possible for a product to score well in the test by having e.g. very good behavioral protection, but a weak URL blocker. However, we would recommend that all parts of a product should be as effective as possible. It should be borne in mind that not all malware enters computer systems via the Internet, and that e.g. a URL blocker is ineffective against malware introduced to a PC via a USB flash drive or over the local area network." @Rasheed187: I think in your case you are more interested in the malware protection test, which does not consider the URL vector.
Thanks for the extra info. However, now that I think of it, then what do they mean with ''compromised''? I mean who cares about how malware ends up on the system, the only job of AV's is to block the malware from running both pre on post execution. So did these AV's fail to block malicious URL's or fail to block the malware from running?
Are you still there? What about my latest question? Seriously, why do these malware tests have to suck so hard? Just make it more clear if they failed to block malware from running or failed to block malicious URLs from running. But then the next question is, who cares about malicious URLs as long as they eventually blocked malware from running?
Paranoia is good for sales of AV products. You are right: If the malware can't run, it can't hurt you. And, my go to defense is a daily image backup. I really don't care that much (anymore) about all of it.
'<SYSTEM32>\cmd.exe' /C netsh advfirewall firewall delete rule name=all program=<Full path to file> '<SYSTEM32>\netsh.exe' advfirewall firewall delete rule name=all program=<Full path to file> '<SYSTEM32>\cmd.exe' /C netsh advfirewall firewall delete rule name=all localport=1688 '<SYSTEM32>\netsh.exe' advfirewall firewall delete rule name=all localport=1688 '<SYSTEM32>\cmd.exe' /C netsh advfirewall firewall add rule name=AutoKMS dir=in program=<Full path to file> localport=1688 protocol=TCP action=allow remoteip=any '<SYSTEM32>\cmd.exe' /C netsh advfirewall firewall add rule name=AutoKMS dir=out program=<Full path to file> localport=1688 protocol=TCP action=allow remoteip=any '<SYSTEM32>\cmd.exe' /C netsh advfirewall firewall delete rule name=AutoKMS '<SYSTEM32>\netsh.exe' advfirewall firewall delete rule name=AutoKMS
Well, I don't really care about paranoia, but why do these tests have to be so complex, you know what I mean? Just make it clear how many malware samples each AV failed to protect against, without all of this mumbo jumbo.
Complexity makes it appear more thorough to some- but of course we just need to know the basics as you stated. What blocked and what didn't? Its really very simple.
Yes exactly, this is my point. Looks like these professional AV testers haven't heard from the phrase: keep it simple stupid.
I do like to see some BB tests, some sort of it. Online (no cloud lookup) and offline BB test. I know, it's really difficult to do BB test, because many BB works differently. Some security software BBs are more towards to HIPS like, and some can analyze malicious behavior more deeply and if possible can revert the damage done by the malware(s). Some Security Softwares and their BB component does the online lookup, but while doing so, the mighty malware is running (encrypting) your file in the background and after the "cloud lookup" the verdict "this is a malware". But all the files meanwhile are encrypted. Sure it stops the processes, but its too late. Yes, GNU is not Unix and BB is not HIPS
Perhaps you should post the result or your testing, or make a video on YouTube? I have been watching a couple of ''amateur'' malware testing videos on YouTube and it was kinda shocking to see how many AV's fail to protect against ransomware. That's why I decided to add NeuShield to my setup, which is of course also not bulletproof.
But its difficult to test, because some softwares relies on cloud lookup and others are more hips based ones. Sure i can do the test with several "different" malwares. And only BB component activated(if awailable). Offline or Online test. Something like Formfook variants vs offline BB. Interesting enough, trialling K7 AV, and it works quite good against newest bazaar samples, so far. As for the best what i've tested is McAfee Endpoint and its Dynamic Application Control aka "Sandbox" of it with it's RealProtect BB is the best i've ever used. By default it's home version of McAfee. Its so damn heavy on the system. I disabled all but its DAC and its Realprotect (GTI sense high), and with bazaar samples i throw against, i caught them all. Suspect! etc. So my opinion, the best it McAfee DAC(sandbox)/RealProtect COMPONENT.
