AVLab - "Test of software for online banking protection"

https://avlab.pl/test-software-online-banking-protection
Full report
https://avlab.pl/PDF_avlab/AVLab-Test-of-software-for-online-banking-protection.pdf

 

Interesting, but it doesn't explain how it was tested? Was this tested with real life malware or simulator? And I noticed that SpyShelter was the only who passed all tests. But here is the problem, SpyShelter doesn't auto-block most of the stuff that was tested. So I would love to know which alerts were presented. And all of the other solutions do try to auto-block these attacks, with behavior blocker or safe browser.

 

You didn't notice Norton?

 

Kaspersky and possibly Sophos Intercept(rebagged HMP-A + Sophos IS), can't remember also received perfect scores. -EDIT- Also Comodo and ZoneAlarm passed all the tests w/"banking mode" settings.

Appears to me this test combines elements of MRG banking test plus things SpyShelter's own test tool detects such as clipboard scrapping, etc.. So I would except it to do very well on the test.

 

IDK

 

In other words, they "cranked up" SpyShelter's HIPS to full alert mode. Which brings up the applicability of actual use. Anyone who has used SpyShelter in this mode will in short order reconfigure it to something one can live with on a daily basis.

 

I have to question the validity of the reported results when the description of the Panda Dome Advanced section refers to Kaspersky and appears to be a copy of their results. Maybe an editing error but if they can't get that right why should any credibility be given to the rest of the report?

 

Spyshelter got perfect score both in default and modified settings.

 

The last time I tested SpyShelter Anti-Keylogger at default settings, there really was no reason to configure further. The alerts were for my purposes unbearable.

-EDIT- The also brings up the "usability" factor in this test. It appears that wasn't addressed in this test in regards to "hardened" settings. The average PC user is not qualified to be answering a barrage of alerts from his security product.

 

The one test result I am having an issue with is Norton's. It had a perfect score in both test modes. I really can't believe it has improved that much.

Also note that in many of the products, the firewall was set to Interactive mode.

 

Standalone HIPS are not made Average Joe, it is niche product for security geeks.
All "usable" HIPS are coupled with cloud lookup, whitelisting, and whatever they can to reduce prompts which obviously reduce its full power.

I use Spyshelter in "Ask user mode" so I answer dozen of prompts when running softs for the first time or when updating them.
For some LOLbins, I set SpS to never remember my choice.

HIPS are made for full control, by not doing so, you defeat its purpose.

 

My bad, I guess I had a bad day yesterday, they also explained how it was tested LOL. So looks like Norton, Panda, ZoneAlarm, SpyShelter, Norton, Kaspersky, and Sophos Intercept X all passed when banking protection was enabled. And it was tested with a simulator, I'd rather see them tested against real banking trojans.

The thing is, all other tools try to block these attack techniques automatically. I'm guessing that SS can only block test 3, 5, 7 and 8 automatically with the keystroke encryption and anti-SSL sniffing feature. For all others it will present alerts. I'm surprised it could block all DLL injection methods. Would like to see more proof for that, I wonder which alerts were presented.

 

Based on this:

the only simulator tool used was the Bettercap tool mentioned for MITM test.

His reference to use Python scripts leads me to believe this is how he deployed the malware used in the test. In other words, it was real malware samples although not deployed as they were originally in the wild. This could possibly adversely affect the Next Gen products whose ML algorithms are dependent on that behavior for a probability exceeding threshold positive value. Although, it appears no Next Gens were included in this test.

 

I'm sorry, but I can't visualize it. What would make sense to me is that they simply run a simulator or real banking trojan on a system and see how tools react. Of course, AV's may be able to detect it as malicious, but I believe the goal was to test the pro-active protection from all of these tools.

So in this scenario, malware is already running in memory and trying to perform all of those tests. But now that I think of it, how was Win Defender not able to block it? Probably because the attack was simulated in a different way, would like to have some more info about this.

 

The use of Python scripts in malware testing is nothing "new or suspicious." The PC Security Channel: https://www.thepcsecuritychannel.com/ , has used then for some time in its security product testing. I also suspect they are used by the major AV labs. Simply put, they are a way of automating the test procedure. However, they are not appropriate for all malware testing such as ransomware.

 

I never said they were, but I don't understand exactly how they are being used. So in order to simulate such an attack, do you need to click on some script file? Like I said, to me it would make more sense to simulate a real life attack.

So let's say you're browsing, and you get to see on some site that your PC is infected, so you will now download a fake AV. You will ignore SmartScreen and UAC, because you really think you're infected. This fake AV is zero day, so your AV won't detect it. So now the question, which tool can block this fake AV from stealing money and data? Some will use the safe browser, and others will use behavior blocking.

 

Here's how the PC Security Channel explains its Python script usage:

Does this in any way mirror how the malware payload was actually delivered in the real world? Of course it doesn't. Note that in the realtime tests by AV-C and AV-Test, they actually download the malware from known malicious URLs which is a more realistic test. However, it excludes other malware delivery methods such as via an e-mail client and the like.

Bottom line - automated testing bypasses AV products front-end protections such as web filtering which employ among other things, IP blacklisting and the like. In other words, tests like this only evaluate malware protection after the malware is resident on the disk. The whole emphasis of modern AV protection is to prevent the download from occurring in the first place.

 

so true.
and the same goes for sw like sap and vs, whereas ag and nvt erp provide rock solid protection for the geeks who know what they're doing.

 

Another simulated test that is supposed to prove something that really proves nothing.

Odd that McAfee didn't participate. Makes you wonder why? Don't see Malwarebytes, Zemana, Hitman Pro and a few others either.

Never heard of mks_vir even though it goes back 32 years!

 

Spy Shelter... not bad.

Anyone think the results would be the same for Premium given its Internet Security feature versus Firewall with its additional two-way rules/trusts Firewall Protection and anti-executable Application Execution Control?

For recall:
https://www.spyshelter.com/internet-security/ (module details)
https://www.spyshelter.com/download-spyshelter/ (comparison chart)

My only hands-on was a ways back with the now defunct Free version.

 

Well, if you're not able to download malware because of web filtering, then how the heck will you be able to test behavior blocking capabilities? That's why you should simply assume that people are somehow being tricked into running malware. So you should run malware and see how AV's react. If they can't stop it via signature/heuristics, they should be able to block it via behavior blocking or safe banking browser.

 

Ah ….., "the chicken or the egg" scenario. But most appropriately, the applicability of the truism "an once of prevention, is worth a pound of cure."

Simply put, one doesn't have to worry about process execution risks if there is no process to execute in the first place . The purpose of AV Labs is not to exclusively evaluate post execution blocking capability, but all malware blocking capability.

 

I think it is more to make money and to provide advertising fodder for those products that score well.

To be sure, I am not a fan of big pharma but how do we know if a particular drug is good enough? The best and most conclusive way is to conduct "real-world" studies with 10s of 1000s (or more) patients out in the real world in real-world scenarios, monitoring them over extended periods of time to see how well the drug does.

Because such a study adapted to computers is impossible (not to mention prohibitively expensive for such test laboratories), these simulated tests in laboratory environments are just that, simulated and not in the real world. It is not the same. These labs remind me of those who try to guess what should go into annual flu shots. Did you know the flu shot is considered a good match if ~50% effective!

Of course they are guessing for the next flu season so the analogy is hardly perfect. But the antimalware testing labs also throw in 1000s and 1000s of different malware at these products even though 1000s have not been seen in the wild for many years, or they only affect XP or other long superseded and obsolete operating systems. Then they rate the program on how well they do against that malware too. Is that realistic or truly indicative of actual threat seen today in the real world? No.

I am not blaming the labs, they don't have the resources to monitor, in real time, on a massive global scale, exactly what malicious code is being a threat today. That would take a company that has a massive global reach, the deep pockets to monitor and continually analyze the current real-world threat, and the initiative to rid the world of malware to do that - a company like Microsoft, the only company in that list of antimalware makers who would love to see malware go away - and is actually doing that sort of research as we speak!

Note all the other companies need malware to thrive for them to stay in business. If malware goes away, so do those other companies. This is exactly why Microsoft does NOT code their program to score well on simulated tests. They don't need the advertising fodder to generate revenue, or to entice consumers to choose their product over the competition.

I am not saying Windows Defender is the greatest solution out there. It clearly is not. But is it good enough? Of course it is! At least with Windows 10. If not, there would be millions and millions of infected W10 systems out there, with more and more getting infected every day. And that is not happening. Yet many sure want us to believe it is. Makes you wonder, why?

I mean look at where the link in this thread's opening post takes you. What is the one and only product displayed in that introduction? But NOOOOO! That is not a biased report. Yeah right.

 

Exactly, but in a test you can never completely simulate real life. Because you can never know how people will react. In the scenario that I described, people will simply ignore warnings from SmartScreen, Safe Browsing and perhaps UAC. You can always be tricked into running malware, think of tech support scams. And don't forget about the so called "supply chain" attacks on CCleaner and now apparently also on ASUS Live Update.

https://www.zdnet.com/article/suppl...s-through-hijacked-asus-live-update-software/

 

Those people include the bad guys.