Comodo Leak Test & paid VS free security

Hi guys
I've been messing around with the Comodo Leak Test and I'm wondering to what degree you can verify a security programs effectiveness based on the result of the test.

Here is a few of my results. The results of the test go up to a maximum of 340 points.

WITHOUT any security program installed:
XP 32bit: 20 point
Vista 32bit: 120 point
7 32bit: 150 point
7 64bit: 190 point

Result with mentioned security software installed (generally with settings at max and whitelists disabled):
ZoneAlarm Free, XP 32bit: 40 point
Comodo Firewall, 7 64bit: 340 point
Online Armor, XP 32bit: 300 point
PC Tools Firewall Plus, Vista 32bit: 250 point
Bullguard, Vista 64bit (default configuration): 190 point
Kaspersky Internet Security, 7 64bit: 190 point

Isn't the HIPS in KIS supposed to be among the very best? If so, how come it does not perform any better in the Comodo Leak Test? After all, its a paid program...

Seems to me that you can actually get a higher level of security with free programs than with paid programs (at least if you can live with the pop-ups).

Any comments?

 

I think that the controversy between free and pay security programs is a false problem. There are naturally differences between the free and the pay version of the same software; but if we compare different softwares, the reliability and the safety of a security software have to be evaluated considering his functions and his features, and tested sure with serious tests, personal too, but especially on the wild. So, all depends from which programs and in which phase of their development we consider, not from to be free or pay.

 

Thanks for the comment. However, I'm still not sure how to explain the fact that a well-respected and paid security program Kaspersky Internet Security doesn't perform better, while several free programs are top-notch in the test.

This is my theory so far:

For a security program to raise the security as high as possible WITHOUT giving the user any pop-ups, it can't go over a certain limit. It just can't always be decided automatically, and in the background, whether an action is legitimate or malicious. Putting the "safety-bar" too high will prevent legitimate programs from running and the average user will get pissed. Since paid programs are primarily made for the average user, they are designed for security up to the level where the pop-ups starts to become a requirement, but no further.
Free programs like Online Armor, Comodo Firewall, PC Tools Firewall etc. are made for the user who already know what he is doing. They can provide better security since they can handle all the situations where paid programs can't decide what to do and therefore allows the action. They handle these situations with a pop-up asking the user.

My conclusion is that paid security programs generally are more userfriendly, and perhaps a good choice for the average user. But IF you are a somewhat "advanced" user and don't mind getting pop-ups, you can save your money and at the same time get a higher level of security by using a good free application-firewall or similar.

So paid program are more userfriendly and free programs can provide better security for the advanced user.

Comments?

 

In KIS, go to Settings -> General Settings and uncheck "Select Action Automatically". With Interactive Mode (that checkbox unchecked) I get 310/340 on 32-bit XP.

Some of the tests (like KnownDLLs) are failed because they use non-malicious MS signed files, which Kaspersky has whitelisted. If real malware would use those technigues, KIS would block it.

At default settings, this is the case with HIPS programs. However, you can make many paid programs just as effective as the free programs by changing their settings. For example, in default settings KIS decides what to allow and what to block automatically based on whitelisting and heuristic analysis results, because the average users couldn't answer a single pop-up. However by unchecking that one check box you get pop-ups just like with the free programs.

 

Against who's yardstick? Many people view leaktests such as the Comodo one as worthless, attempting to test HIPS against scenarios that have no real bearing to real world malware. You've taken a bunch of potentially meaningless tests, run them against some suites and reached an implausible conclusion about the worth of paid versus free security software.

Even if malware did use some of the techniques in the CLT, any good anti-executable would stop the malware before it launched.

 

I always wonder about the usability of anti-executables. I'd imagaine any AE to be even harder to use for a home user than a quiet HIPS.

I think some malware may use some techniques which CLT also uses (can't remember exactly what tests CLT contains again). Also, why not act proactively and block the techniques before there's some malware ITW using them? Blocking those techniques can't cause any harm, can it? Sure, the tests use many useless technigues which malware will never use, but I'm sure there are also useful tests in CLT and other similar testing tools.

 

Sure there may well be, but they are not a basis to judge the effectiveness of HIPS in the manner suggested here. The effectiveness of a HIPS is only as good as the capabilities of the user. If the user is going to launch malware then it doesn't matter a stuff how well their HIPS performs in leak tests.

 

Thank you Rampastein, I have followed your suggestion regarding KIS and disabled the option for automatic selection (as mentioned, I had already turned of whitelisting/digital signatures). However I still do not get more than a single pop-up when I run the scan, and I still only get 190 points.

It seemed strange, but I came to think of PC Tools Firewall Plus which apparently only have the full HIPS protection on 32 bit systems. PC Tools are quiet about it, but feel free to check for yourself. Could the same be the case for KIS?

I installed KIS on a Windows 7 32-bit and ran the test with the same settings as before. This time I got lots of pop-ups and a result of no less than 310 points! So same result as Rampastein got.

It looks like you need a 32 bit OS to get full protection from KIS. I wonder how many of their users realize that?

So an advanced who that wants the ultimate protection and is running a 64-bit OS should apparently stay clear of KIS (and as mentioned also PC Tools Firewall). And if he is running a 32-bit OS there is still no need to use a paid program as he can get the same security free, and even without sacrificing any convenience (he will get the pop-ups in either case).

So until now it still makes sense to conclude that while the easy-to-use paid program may have advantages for the average user, the advanced user might as well stay with the free programs.

 

That's surprising. I don't have a 64-bit system so I could check it myself, but from what I've heard from some KIS beta testers who have 64-bit systems I've thought that the HIPS on 64-bit systems, while not as strong as in 32-bit systems, is still very good on 64bit. 190 points seems to be the same score as with W7x64 without any HIPS installed. What tests did KIS fail?

Yes, there are some differences between some programs though, and what the advanced user chooses to use often depends on the user's personal preferences and choices. I like having everything in one single application/suite (which wouldn't be possible with the current free programs) and I find the application whitelist in KIS much larger than in any free HIPS programs (which is one of the most important reasons why I like it so much). However, I didn't pay for KIS, instead I got the license for free for beta testing actively. If I wouldn't have got that license I would be using something else at the moment (while still testing KIS though).

 

Check if CLT is in Application control Trusted group (it's trusted in KSN). Move it to Low restricted if it is.

 

@Rampastein
Please see the screenshots. They will show you which tests KIS missed.

By the way, you can actually get security suites for free. I would recommend Comodo Internet Security, but there are other options as well. However, as you've mentioned, their whitelists may not be as large as with some paid programs. I don't know about that as I prefer till disable all default whitelists.

@3x0gR13N
I've followed your suggestions and moved CLT to low restriction in KIS. I'm now getting the result of 300 points.

Honestly though, I'm not sure if this gives a more precise result of the HIPS capabilities in KIS against actual malware. Any thoughts?

 

Most free suites don't have as many features as paid ones and some of their parts are weak. For example, Comodo doesn't have a web traffic scanner (which I like to have, even though it wouldn't be so necessary) and their AV component isn't known for it's efficiency.

I don't know how you get so bad results with CLT on x64 though. I've seen on KL beta testing forums that on x64 KIS should block almost all of the tests (like on 32-bit systems). For example, you failed RawDisk, but here one tester posts a bug report related to the HIPS: http://forum.kaspersky.com/index.php?s=&showtopic=164848&view=findpost&p=1343720 (post #26) You can see in the second screenshot that KIS blocks direct disk access. With some googling I could find many more examples of KIS blocking CLT tests on x64.

Are you using a real machine or a virtual machine to test? As far as I know KIS doesn't work correctly on some VMs.

 

I'm using VMWare for most of my tests. I have never really heard of any issues or incorrect results caused by that.

You are right that the Comodo antivirus is not known to be the best, but some tests seem to indicate that it has approved a lot. I remember a youtube video where Matt from remove-malware.com wanted to test the HIPS in Comodo Internet Security, Defence+. It turned out be be pretty hard because the antivirus caught almost everything.
In any case, the primary security would be in the Defence+ HIPS, not really in the antivirus, at least for the advanced user.

As long as all files are scanned before execution, what would be the benefit a having a web traffic scanner?

 

I'm not that familiar with the subject. I simply like it when my AV blocks a malicious site and replaces it with a custom HTML even before anything is downloaded though.

I don't plan on switching from KIS as long as my license is active as it just has a perfect set of features and configuration options (for me). It's application whitelist is quite large, at best it has had some unofficial game modding tools (for ARMA 2) made by the game's fans in it's HIPS whitelist.

For other advanced users, who don't want to pay and who can't get a license for free, CIS is a good choice. Currently I'd recommend NIS for average users who are able to pay, KIS for advanced users who are able to pay (or get a free license), and programs like CFW, OA or Sandboxie for advanced users who don't want to pay. As for average users who can't pay, I don't think there's much better than a normal free AV, as things like HIPS, Virtualization etc. are often too complex for them.

I'm out of ideas with your CLT leaktest results with KIS though. I'm 100% sure that if KIS would be functioning normally it's score would be much better though. With 190/340 it looks like the HIPS isn't functioning at all or something.

 

I could use another physical computer, another installation of VMWare, another virtual installation of Windows 7 64-bit, another installation of KIS and see if I can recreate the issue.
I have a feeling that it won't change the result from the leak test much, but it would take some time and I guess the result can still be dismissed if somehow testing in a virtual environment should not give reliable results. But if there is "public demand" I'll see if I can find time for the test. Let me know!


Anyone out there using a traditional paid "internet security"-program that scores well in Comodo Leak Test?

If so, have you changes the default settings so that you are now getting pop-ups like with the free application firewalls?

 

As far as I know the KL devs also use VMWare so it shouldn't matter. Although as you can see in post #12 your results are strange.

 

Wasn't KIS low score already answered?

 

Ah what.. I've seemed to miss that line
Nevermind about the "low" score with CLT then.

 

To some extend yes, but does it give a more reliable test result that you in the case of KIS change the trust-level for the Comodo Leak Test? I had to do that afterwards, knowing what file I have to change the trust-level for. You won't get such a "second chance" with actual malware...
Any thoughts on this?

I just had a system with Comodo Firewall where I tried to download the leak test from grc.com. Comodo Firewall warned me about the leak test before it even downloaded. Apparently the latest version of Comodo Firewall (and CIS I'm sure) have added a "Cloud scanner". It also reacts to EICAR. I assume this cloud scanner does the same thing as a web traffic scanner.
So if one day you happen to become really poor and your license for KIS expires...

 

In case of CLT, it has been around for quite some time and has built up reputation, and doesn't show any (truly) malicious behavior hence it is Trusted in KSN.
As they say "it's just a PoC", so it doesn't reflect how the product copes with actual undetected malware.

 

As 3x0gR13N said CLT has existed for a long time, has many users on KSN Cloud and has been added to the whitelist because of that (automatically I guess). Malware won't get added to the whitelist because a single malware sample doesn't usually infect many computers (= a low number of users, only a dozen at most), and a sample doesn't exist long before it gets deleted from the users' computers.

Malware will usually get also a bad "danger level" from heuristic checking (also affecting reputation) unlike CLT which gets a low danger level because it doesn't really do anything malicious.

Since the malware isn't in the whitelist, KIS puts the malware to the Low/High restricted / Untrusted groups (depending on heuristic scanning results), and in Interactive Mode you get prompts when the malware tries to do it's malicious stuff.

 

Thank you both for your comments. What you write makes a lot of sense but there is still one thing I don't get.

CLT was treated as "safe" because it was on a whitelist i KIS, right? But I had that whitelisting disabled during the test...

 

Did you uncheck all three options "Trust applications with Digital signature", "Load rules for applications from Kaspersky Security Network" and "Update rules for previously unknown applications from KSN" before launching CLT for the first time? KIS puts the applications to their groups the first time they're started, and with that one option (update rules for unknown apps from KSN) enabled KIS can change applications' permissions without any user interaction (and in this case it moves CLT to Trusted group once it's launched).

Just tested it, works correctly for me if I uncheck all those three options.

 

Rampastein: Those 3 options are exactly the ones I have unchecked. In the Proactive Defence tab I have also unchecked "Applications with digital signature" and "Trusted in KSN database".

You are right that CLT gets automatically recognized by KIS and added to the database as a "Trusted" program as soon as it is executed the first time. The wierd thing is that is is treated as a trusted program even with all the mentioned settings/whitelists unchecked.

I have now installed KIS on completely different hardware, at a different physical location, on another virtual Windows 7 64-bit (still using VMWare) and chosen the same settings as above: Almost the same result, this time my score was 200 points.

It looks to me like KIS ignores my attempt to disable the whitelist and still treats CLT as a trusted application. I guess this is only an issue for users wanting to completely turn of all whitelisting, which only makes sense for a few security-nutcases like myself, but if it's a bug I'll consider reporting it to Kaspersky.