Six HIPS Programs Reviewed and Rated

Yes version 2 will be much better!

dja2k

 

"Wait for version #x" -- sounds a bit like the old Brooklyn Dodgers' lament: "Wait till next year."

Ohhh, dem bums!

 

I haven't had a chance to properly review this yet, but at first glance my main concern for me was the failure of test 3 in hostile web browsing. I'll be contacting the author to find out what happened there.

I'm not concerned where the tests about protecting registry keys or kernel mode keyloggers - because I already know where OA protects, and those that it does not. That's just a marketing and add-to-next version problem.

I'd be very interested to run the next version of OA against some of those tests and see how they went, especially the version with the driver

I'll keep out of this thread now - as Bellgamin points out this sorta thread isn't really for vendors.

 

Pitching a Sandbox with HIPS apps in a test basically invalidates the test. Waste of time. Either test sandbox apps collectively or HIPS apps collectively, but you can't do a test with one Sandbox and all the others are HIPS. If the author of these tests strikes the results for DW off the page then it may be worthy of consideration. But as it is? There's one word that qualifies the test as it is...Misleading.

muf

 

Not necessarily. The way i see it he just tested 5 recent security apps. that reduced the noise of pop-ups in their own way, which he already had stated were the downside of HIPS, and i agree. A normal user doesn't want that, but would probably want the protection.
Kees1958: i know what a sandbox is. I have one. What i didn't understand was the difference between DefenseWall HIPS and DefensePlus.

 

Both are lifetime?

 

Well, SocketShield differs from traditional AV/AT/AS and also differs from HIPS but I think that SS is a kind of "smart antiexploit" so its concept is nearer to HIPS than AV/AT/AS. Also, considering that part of the test involves drive-by downloads it seems to me right to include it

Yeah, but with different methodology. Only GreenBorder and Sandboxie score high

I feel Gizmo´s tests flawed here and there
Congrats to DW developers

 

The tests are a hodge-podge with no homogeneity; hence validity is limited at best. Should test sandboxes with sandboxes & HIPS with HIPS. As noted earlier, if they're going to include a sandbox why not also include ShadowUser?

 

@Bellgamin
Despite misgivings about methodologies; you must be pleased with CH?
You still running that or moving to DW a bit?

Unfortunately, PrevX: =3rd. (Iknow , Iknow Not HIPS, still... )

Regards.

PS the same tester did do some sandbox testing: http://www.techsupportalert.com/security_virtualization.htm

 

If Gizmo wanted to show the users how different software protect and detect against those specific threats I think its OK to mix different kind of protection-software, but I would have liked to see at least one more sandbox-type of software in the test.

Best Regards

 

A sandbox is not a sandbox!

Confusion


Ian "Gizmo" Richards states that "This blurring and confusion between HIPS and other security products is clearly seen in all the products I tested. Although they are all notionally HIPS programs they are in fact, as different as they similar"

The tests causing much discussing on this forum
http://www.av-comparatives.org/seiten/ergebnisse/HIPS-BB-SB.pdf
http://www.techsupportalert.com/security_HIPS.htm
http://www.techsupportalert.com/security_virtualization.htm

Cause

Using the popolar terms sandbox/virtualisation leads to confusion (see below).

There are three type security programs which are called Sandbox/Virtualisation:
1. Access Policy management.
GeSWall and DefenseWall are examples of this kind of programs. Technically they apply a software restriction policy to system resources. The goal is to obtain OS integrity and Data confidentiality in a seamless way for the user (system works transparent, invisible to the user). So when Ilya (of DefenseWall) wants his program to be classified as a HIPS, he is right. It does protect the host integrity from being 'intruded' or violated by Untrusted programs. Brian (of GeSWall) has struggled to position his product a bit, now his website is also starting to call GeSWall an intrusion protection program.

2. Apply Software restriction AND a virtualise the file system.
On the website of Sandboxie this is explained. Sandboxie and f.i. BufferZone seperate untrused programs with a layer between the untrusted application and your data. These programs are offten called Sandboxes. So trusted programs are allowed to directly write to your harddisk and vulnarable OS-files, the untrusted ones access (write to) the layer in between.

3. Applying also OS virtualisation
With these products the seperated environment not only applies to the file system, but also to the OS-system. This type of software is called virtualisation (like VM Ware). These types of programs offer a Virtual Machine on the same PC (hardware). It is a technique which dates back from the mainframes, but is very effective (Gizzmo performs most of his test in a VMWare environment). Only downside is you often need a second OS (lisence) for the virtual environment.

Test from the user point of view

I think we should stop testing from the technical point of view and start testing from the users point of view, again "Gizmo" states " Does it matter? No, they are simply products that offer another layer of protection for your PC in concert with other security products such as anti-virus scanners.".

My conclusion: security applications have made a good step in making their aps more user friendly.

User friendly approaches are (besides old AV-blacklist approach):
- active heuristics (available in good paid AV's)
- CIPS intelligence based behavior blocking (Cyberhawk)
- CIPS intelligence based whitelist (PrevX)
- policy restriction at threat gates (DefenseWall)

So from a usability point of view some new products are great (the ones which do not confrontate a security novice with questions he/she can not answer).

For me the leaders of this pack are:
1. DefenseWall (great protection, no pop-ups)
2. CyberHawk (free behavior CIPS, only pop-ups when anomalies occur)
3. PrevX (free until you need them, still uses the weakest link, the user, as a last line of defense by asking to run it when CIPS white/blacklist and behavior protection of PrevX do not provide the good/bad answer)

I am interested how the new kid on the block DataSentry will perform (also CIPS, but focussed on data protection by allowing only good/known aps to write data).


Regards Kees

 

@Kees

.Agree, from the end users POV fine terminological inexactitudes mean little compared to either notional or proven performance.
With so much overlap now HIPS/CIPS/Application filtering/Sandboxing/ it may be increasingly difficult to provide standardised testing methodology.

From my level as cellar dweller I just want to know what works and will tool around looking for test info from as may sources as possible b4 deciding what to trial, then come here looking for insights.

Also agree that, obviously, full virtualisation is a different fish all together.

Regards.

 

It wouldn't pass any leaktests, because that's not what it's meant to do. Those files have to be downloaded before the test can begin, so it would get a "fail" on all of them, which wouldn't be fair to the vendor. To include SocketShield in a test of leaktests would be like including a router in the same test, or like including SpywareBlaster in an on-demand anti-spyware test - the test would be completely out of scope of what the product actually does. Then, of course, it would cause a big stir about how ineffective it is because of all the things it doesn't do, confusing the fact that it's meant to stop drive-by downloads and nothing more. What happens after something is sucessfully downloaded is left to the rest of your defenses.

You could include DriveSentry, since it's technically a HIPS, and it wold also fail most of them because it's made to stop malware at a completely different point; which is when it's first written to the drive. Testing leaktests against it would be pointless as the purpose of the program is to prevent the files from being able to infect the system at all, not stop memory events after it's already in the process of infection.

You could also include DeepFreeze, ShadowUser, and FD-ISR, since they are generally included in virutalization products and claim to handle any kind of malware. Obviously they too would fail all leaktests.

This is why I tend to not like these tests using leaktests. They give a very static view of features that may or may not test what the program is meant to do. In any infection there's a long series of events that occur, with leaktests only testing the very late stages. Imagine a count of 100 steps a piece of malware takes; the leaktests would test steps 75 to 85 (approximately). If a program is meant to deal with the malware at any point outside of that limited range, a blanket "FAIL" score is given to that program without ever looking at how that program actually approaches the problem. If some product is meant to stop the first 50, and those 50 are something common to a much wider range of malware, would that app deserve the "FAIL" that it would inevitably get? That kind of test also doesn't give you any idea of what it would be like to encounter the malware in a real-life situation, and whether you would actually be likely to allow it or stop it if applicable.

I will say, however, that this test was acutally probably pretty good for the likes of DefenseWall, as those actions are a good part of what DW is meant to block and it gives you at least some idea of how it actually works. It doesn't give you the full picture, but it gives a good idea of what it might block that you might not otherwise know.

 

Notok,

I agree about leak test and the fuzz about outbound traffic protection. This is like a bank manager being more concerned on how the thiefs might run, than protecting the money theft.

I think you understood the drive by downloads wrong. There are a few notorious sites which offer cracks/key generators, but put really damaging malware on your computer (see example, please anyone do not try this at home google warns for the site).

The drive by infection is about visiting web sites and getting virusses and all sorts of damaging mal-ware without you having okay-ed that. (like dfk threatsimulator).

 

No. DefenseWall's updates are not lifetime now.

 

To illustrate what malware is trying to get it's way to your computer by just driving by I have also added a picture of the DefenseWall log (processes, registry changes, file downloads, et cetera). This is before you can press the download button (which I did not do off course).

 

I couldn´t say it better
I´m really worried by the great diferences between DefenseWall and GeSWall in these tests using both apps. almost the same concept. I can´t accept that GeSWall is a clearly inferior product only for being free

 

Notok, Lucas,

I think the part on which GeSwall failed in the other test was a drive by and shoot in the foot test:

*** Quote
The site I used, a Russian cracked software site, uses flaws in Windows and Internet Explorer to download malware without any user action or knowledge. Typical exploits include the well known iFrame and WMF exploits though the sites will repeatedly try a sequence of exploits if not initially successful. If finally successful, the sites download multiple malware products, often running into tens of megabytes.

Four of the eight sandbox products failed this first test. That is, some of the malware products were able to infect the PC outside the sandbox The products that failed this test were: Altiris SVS, GeSWall, VELite and Virtual Sandbox.

*** end quote

Although not certain Gizmo seems to have performed the same test on DefenseWall

*** Quote
The five usage tests were:

1. A keylogger test involving the simulation of four different logging techniques plus the installation of four commercial keyloggers

2. Installation of the DFK security test program. This is a sophisticated blended threat simulation that disables defenses, bypasses firewalls, installs a cleverly disguised trojan, a virus and a keylogger all masked with a rootkit.

3. Hostile browsing tests using three different drive-by download sites

4. A shoot-in-the-foot test involving the installation of an infected game, screensaver, keygen, crack and a search toolbar. All were obtained from currently operating web sites.

5. A rootkit installation test using Hacker Defender and FuTo

*** End quote

I use DefenseWall on my wife´s PC and GeSWall on my laptop and am really happy with GeSWall. Because GeSWall failed those drive by test, I made the previous screenprints on my wife´s PC (with DefenseWall). The only difference I know of is that GeSWall uses the Microsoft group policy management. Maybe this is the reason for different performance (using MickeySoft is always trickey).

 

I loooove DeepFreeze. I THINK your comment [quoted here] means that DeepFreeze would allow the leaktest to call home (or wherever). Correct?

Further, I assume that "virtualization products" would also be useless for protection against keyloggers. Correct?

 

My point was that one has to consider the scope of what products do.. HOW they protect you, not just whether it blocks this or that leaktest.

There's a lot of steps necessay for malware to infect your system, leaktests are meant to test a very small number of those steps, and only later on in the infection process. Any application that handles malware outside of that limited scope would be made to look ineffective if all you're doing is tallying what actions were and were not blocked.

Point was that this test was primarily a test of behavior blocking. Although it did have a small portion for drive-by-downloads, a script filtering product such as SocketShield would not belong in such a test. It would only serve to confuse the purpose of SocketShield and make it appear ineffective to those that consider leaktests a measure of effectiveness of a security application. ANY program tested outside of it's intended scope of protection will do poorly. Each product tackles malware in a different way. Test files aren't malware, they are meant to test very specific things. Your anti-virus wouldn't do well either unless they decided to add detection for the test file, which would speak absolutely nothing about that program's effectiveness against malware in the wild.

These tests are fine when testing against a product specifically made to stop the kind of things that leaktests simulate, but if the program is made to handle a different kind of threat, or handle it earlier on in the process, then the test needs to be done within the scope of what that program actually does.

It's just the same as not testing anti-spyware products with an anti-virus test bed... it would make the anti-spyware look quite weak, but that doesn't mean that the anti-spyware application doesn't do what it's meant to do quite well.

It all depends on the virtualization app and how it was made to work.

 

Thank you
I was thinking the same

 

I kind of agree.

Realistically speaking, a sandbox type program would have a big edge in such tests and would probably win most of the time.

With sandboxes anything remotely dangerous is blocked, so it can't even get off the ground.

With classic HIPS, there might be a warning of a driver being installed, and as far as I can see Big G doesn't consider this as a specific enough warning, so he allows it. Then Game over.

Also look at the DFKS tests, all the HIPS detect almost exactly the same thing, but DF is pass ,but the rest are fail because in the prompts "nothing specific" was raised. Of course if the user locked his interface, and auto-block everything then wouldn't they be a pass? DW is not smarter in this respect, it is simply auto-blocking everything....



Of course the two aren't really directly comparable, because if you try running everything in the sandbox, many things wouldn't work and you would have to run it out the sandbox anyway.

But running your browser in a sandbox isn't a bad idea, assuming there isn't any occasional compatibility problems (e.g trying to run updates).

 

Throwing in a sandbox doesn't invalidate any test. A test is a test, you pass or you fail.

Using the results as a "comparison" might not be valid, but it doesn't change the test results.

Did that make sense?

I would suggest printing out the test result chart, and whiting out Defensewall's results -THEN see how well your favorite did

 

Well let's just say, that the winner in this one had an unfair advantage because of the testing methodology and leave it at that. Whether this 'invalidates' the test or not, well ....

I guess my point is that sandboxes aren't necessarily technically superior as some people seem to take this test to show (though they may be) , but because of the way the test was carried out, the sandboxes would always seem better.

The strengths and weaknesses of sandboxes versus other kinds of HIPS cannot be judged by just looking at such tests, because sandboxes will always look better for reasons already mentioned.

 

I think it's just a matter of not taking it for granted that a high or low "score" is in direct relation to how effective a program is within it's own scope of protection. If you read the commentary, you could at least extrapolate some information on how the program works.