Are HIPS programs TRULY effective?

There are now LOTS of HIPS programs for us to choose between. In fact, the *growth industries* with respect to security programs appear to be (1) security suites, and (2) HIPS.

As the number of HIPS programs increases, we are confronted with the dilemma of deciding WHICH ones are *good* ones, and which ones are only so-so. It seems to me that most of the forum threads that discuss this issue are mainly subjective in nature.

Forum threads dealing with which HIPS is *good* usually discuss GUI, speed and adequacy of support, and so forth. But what if a person buys a HIPS that has a superb GUI, from an outfit that provides super fast/friendly tech support but -- unknown to the user -- the HIPS program is more bark than bite when it comes to actual protection?

IMO, forums do not discuss *hard data* concerning HIPS effectiveness for the simple reason that such data are not generally available.

*Firewalls have their leak tests
**Antiviruses and antitrojans have tests by AV-comparatives, et al
***But HIPS are basically untested.

This situation was brought very much to my attention by a post made to a certain HIPS support forum by user "defenestration." That user gave us a link to a PDF download which provides a lengthy technical discussion of how HIPS programs can be potentially penetrated. That download link is THIS one.

A subsequent reply to that post gave a link to some actual HIPS tests over at THERE. Those were absolutely the very first HIPS-test-results I have ever seen -- but then I lead a rather sheltered life so there might be others (I hope).

I have 2 main goals in posting this thread...

1) I hope that the denizens of Wilders will make comments about (a) the links cited above, &/or (b) HIPS testing in general, &/or (c) any methods that they personally use for deciding WHICH HIPS that they will use.

2) I also hope that folks will encourage independent test organizations (such as AV-Comparatives) to become interested in testing HIPS programs.

aloha... bellgamin

 

HIPS is only as effective as the user because if you allow something, no matter what HIPS you're on, you can still get infected.

 

That pdf you mention there Bellgamin, is from 2004. From what I can tell the Author has his own HIPS , coincidentally on a site also not updated since 2004... So, I say the pdf is valid, but should be well covered by all heavy duty HIPS. It seems the Authors HIPS could handle the problems mentioned in the pdf - even in 2004.

I'm settled on the sandbox variety of HIPS, not the Classic variety (pop-up o' plenty type). I think the review you've pointed to answers your topic question, but the all created equal is a good question.

 

Well, none of the trojan tests or leak tests would even run unless you give them permission in the HIPS in the first place, same goes for the real trojans and the weakness the leak tests represent. Isnt that effective enough?
Atleast my HIPS cover more than the execution of the malware (writing to registry, launching applications, injecting code into other processes, trying to get access to internet by injecting code to approved applications and so on) Or maybe I understand the question wrong?

 

HIPS are only as effective as the person using them.

prevx1 would be an interesting HIPS to test tho, since it can differentiate between many good and bad programs.

 

I hope you are correct, but I wonder -- how we can KNOW that a particular HIPS does, in fact, provide such coverage? In fact, that question was the central point of my post.

You have aroused my addiction to buying/trying new security stuff. May I ask: Why do you prefer sandbox variety? Which one(s)?

 

hi bellgamin...

you know that the product have the coverage by testing it.
for file support, just use subst command line.
for registrym search google on registry symlink, there are freeware app that does this for ya. Yet i agree that this is not the thing one would normally do while searching for a product.

About performance, some vendor make many efforts to optimise their product as much as possible. To transform those effort into real number for the customer, benchmark are on their way to release.

 

toadbee gave a link to Ozone, whatever happened to them and their product ? A few well know companys are listed as clients, including the us army

 

Most of the tests and exploits on the first 2 methodology pages were launched from the PC being tested, the test/tool being initially permitted or launched with the HIPS disabled/shut off. Definitely not an examples of real life situations.
Example on page 2 of the methodology:

If the HIPS is being used as it was intended, running all the time instead of being disabled, that rootkit won't get installed. Not unless the user clicks "allow" or the system is infected in the first place. Ideally, the HIPS should not only be running, it should be on a setting where unknowns are just blocked instead of prompting the user, unless we're assuming the system's administrator is allowing these things.

With the PC about to shut down or in the process of doing so, where would this threat come from in a real life scenario? It couldn't be carried out from the web, not like that. The only possibilities are a trojan that had already received those instructions or someone at the keyboard. If it's a trojan, the user either didn't clean the system before installing HIPS (learning mode?) or permitted the trojan, in which case the fault is the users.
If we assume that your PC is physically accessible to a malicious individual with enough skill to use the tools/exploits named in these tests, and call that a legitimate threat, most software security solutions are going to eventually fail. Realistically, how many will ever face such a threat? Many of the tests on those pages couldn't be done any other way. If your PC has data so valuable that someone will go to that extreme, obtaining physical access with a toolbox like that, why is it accessible and why are you using windows?

If we're allowing this as a legitimate threat, what is stopping the individual from bringing a screwdriver and a laptop, hooking the hard drive to his own laptop, and deleting the executables of all the security software, then booting the unit back up? Just how many people are going to access your PC with an external hard drive? On a lot of units, removing the battery resets BIOS passwords. These tests assume too many tools and utilities are able to run or that the user is going to make some incredibly bad decisions or too much unrestricted physical access by someone who knows what they're doing.
I don't have the time or inclination to try and duplicate those tests or an NT based PC to perform them on. When I was testing SSM, I visited malicious sites, tried opening infected e-mail, and attempted to launch a lot of malware and malware installers. As long as SSM was running and used a tight ruleset, none of the malicious code could run unless the user permitted it. I didn't run a resident AV during those tests and don't normally run one now, but I did use a firewall, which I consider a security necessity. I can't conclusively say that SSM or another HIPS will prevent all malicious code from executing. I can only say that it has on my system so far. I still scan everything new that comes into my system. Even if HIPS can replace the resident AV, it's not a replacement for an AV scanner. If we're going to evaluate HIPS, it should be done in a realistic user environment, under realistic conditions, facing the threats users face and properly configured. users other than the one who installed and configured the HIPS should never see a prompt from it. A hacker sitting at the keyboard for an unrestricted amount of time isn't realistic either.
Rick

 

Nice one, herbalist
It sounds really funny: Turn of HIPS, infect the computer with rootkit and say: "Ooh..HIPS doesnt protect as they should"

 

Come on... Can't you distinguish between prevention and detection purposes ?

Among the programs being tested, some only claim to prevent rootkit install (ie. DefenseWall), while some pretend to both prevent and detect it (Prevx, KAv proactive module).

As for the threats starting at shutdown or reboot : Have you never heard of theses malwares hijacking winlogon.exe? I know this can be prevented before it happens, but when you don't, there are chances that the malware is loaded before your security apps (I'm sure one of the malware files did it, during these tests, and I probably still have some screenshots of the hijacking attempt somewhere, but I do not remember which one it was). Anyway, that's not science-fiction as you seems to assume.

nicM

 

HIPS's don't always depend on the user's knowledge. Based on the method taken by the developer of any given HIPS, the end-user can be using a computer for the first time in their life and be super-dooper-secure; although this is not always the case. The absolute best HIPS testing I have seen done is located at http://security.over-blog.com/ (mentioned above).

I plan on reviewing the GUI of a lot of HIPS in the near future, along with basic features. Kareldjag and nicM do a great job of testing the actual security, but sometimes the way they put it together can be confusing (for me at least). I think the actual GUI is very important because it is the 'middle-man' for the user and the actual security, although security is what it all comes down to. Hopefully I will be able to review security products that have already been through some hardcore testing. Personally, I think HIPS are the future and it's always nice to see people researching them before buying.

 

More & more HIPS programs are coming on the market nowadays, each of them claiming to be effective. BUT --are they all EQUALLY effective? If so, then it wouldn't make much difference WHICH one I select to use, would it? Might as well use a freebie or the cheapest HIPS there is and be done with it, right?

However, I think it is rather more likely that some HIPS programs are better than others, and perhaps some are even MUCH better than others. Consider AV-Comparatives tests of antivirus programs. Those tests indicate that some AV programs DO offer better protection than others. Isn't that same situation likely to prevail with respect to HIPS programs?

I may have not made it clear to Herbalist & others that my main purpose was NOT to demean HIPS programs in general, but was (instead) to bemoan the lack of objective test results as pertains to the effectiveness of HIPS programs. More to the point -- I am disappointed that discussions of HIPS programs usually center on such factors as : {1} The GUI is really good (or not). {2} Such-and-such program is user-friendly (or not). {3} Their support forum answers questions really fast. {4} I made a suggestion and they worked on it real fast. {5} I have used this program for (however many) months & my computer is still okay.

No offense, but the replies thus far have not identified any objective information for differentiating between HIPS programs on the basis of effectiveness. I was truly hoping for something better than critiques of the particular links I cited.

Perhaps there is no basis for objective comparison of HIPS programs. If such be the case, then marketing hype and anecdotal opinions will have free reign, I suppose.

 

The fact is they are all different. There are a lot that are almost equally effective, but it comes down to the method you prefer. If you folow the method implemented by the developer than you are usually good to go. Take a look at ProcessGuard and then take a look at DefenseWall. They will both secure your computer with almost the same security. PG allows slightly more security while DefenseWall can be used by people without as much knowledge or just people who get sick of answering questions.

You say you are asing for an actual comparison of HIPS, well you can compare the security, GUI, and support. What else is there to compare?

 

HIPS: Host Intrusion Prevention System. If that testing is based on what HIPS can detect, it's missing the point. The purpose of HIPS is to prevent an intrusion or the installation of malicious code, not so much to detect one after it happens, especially when a well configured HIPS would have prevented the installation and there wouldn't be anything to detect. Those tests would be more for an IDS (Intrusion Detection System) that HIPS software. These approaches are entirely different on the one point that matters most. Prevention is pro-active, detection is reactive. One detects malicious code installed on a system. The other prevents its installation.

Unless the malware was installed before the HIPS, I fail to see when it would have had any opportunity to afterwards. I'll ask the same question here. Which is being tested, the detection of an installed threat or the prevention of its getting installed in the first place?
Bellgamin,
I understand your point. AFAIC, HIPS is what Windows has needed all this time, a way to put an end to the "everything can run and do whatever it wants" problem that's plagued Windows operating systems. The only realistic way I know of to test an app like SSM besides using exploit tests, trojan simulators, etc is to go to the drive-by sites, open infected content, etc. Duplicate what happens to real users with the real thing. I don't have complete records of all the sites and exploits I tried against SSM, but I do have quite a few. Most of the sites are long gone by now, but I did save what malicious content I could. Is this the kind of data you're looking for? Trying to decide which particular HIPS is better would be a problem in itself, as fast as new versions are coming out. Tests of almost any security-ware can be made to say whatever the tester wants it to, just by the choices of malware. There is no truly random sampling of malware or malicious code that can be used that wouldn't be outdated almost as fast as it was released. Setting specific criteria and choosing material for the tests would be critical. The best I can say is this. For all purposes, malware and malicious code are processes. Installers are processes, including those for rootkits. While rootkits can hide from many apps, the installers don't. Any quality HIPS that truly intercepts unknown processes will stop them from ever getting installed. If it can't run, it can't infect. It doesn't matter if it's a common trojan, a boot virus, or a rootkit installer. At some point, some form of an installer, dropper, etc has to put the file(s) there and something has to execute them. How well HIPS software prevents this is a more proper and useful testing of the software's effectiveness.
IMO, the best way to test HIPS software is to see how well it controls processes, not just legitimate software, but any malicious code. Not just whether or not the app/code can run, but what it's allowed to do. Are it's hooks detected and blocked? Is it prevented from modifying the registry? One other thing is certain. Any tests of HIPS software will be just as much a testing of the ruleset as it would be of the software itself.
Rick

 

You might find this interesting bellgamin.
Many of the hijack sites I've posted about have screenshots showing how the programs I use are effective against various real attacks using rootkits / spambots and trojans etc..
http://www.bluetack.co.uk/forums/index.php?showforum=83

http://img.photobucket.com/albums/v472/Hellbound_Designs/Anti-Spyware/file_harvests/proffy209_029.jpg

 

I think it depends on what you mean by effective.

As I see it, there are two ways to test them, one technical/objective, the other still technical but somewhat subjective.

The most straight forward way is to test the quality of the implementation. For example X claims to protect apps from being terminated, so we try various different methods of termination and see how X stands up. Or perhaps trying different ways to startup a program to evade the execution filter/controls.

This types of tests has being done on a limited scale already and is doable barring the high technical level of ability needed for testing. Of course, if tools already exist for testing (E.g APT) then okay, but even then I wouldn't fully trust such tools because they are biased in obvious directions.

In practice though, I don't find such tests that important, because in the real world, for most part people aren't trying to crack these protections because they aren't popular enough yet, so practically all solutions are equal. (How many real world malware have you seen uses the 7 or 8 exotic termination methods in APT).

Another way of testing is to see how HIPS functions in real world situations, by getting infected and see how your HIPS reacts. Here we are testing the scope , how effective your system is at detecting correctly anomalous behavior/states.

I personally find such tests pretty subjective.

Firstly what exactly are you expecting your solution to do?

Is it supposed to protect you even from yourself when you choose to install software packages of your own free will? If so, testing by clicking on malware and letting them run is an okay test methodology.

But if that is the case, the question becomes what do you count as your solution protecting you? What types of prompts count as protecting you? The process starting? (obviously not) . Startup registry changes? (Maybe?), Driver/services being installed? process attempts to terminate? Dll injection?

Or maybe even more download stream effects like attempts to send mail , overwriting x files in y times etc.

The problem is, we cannot state categorically what should definitely be monitored. At least not without going through a through analysis of the common malware threats out there and looking for the smallest effective set.

What I do know is that if you a generous citeria for a 'pass', the solution that monitors the most areas, would naturally appear the best, because they prompt on EVERYTHING.

This is the same as an antivirus solution with a super broad heurstic, so it flags everything. But at least AV tests these days recognise the concept of False positives and try to factor that in.

With HIPS solutions, FPs become even harder to measure. I've seen people who argue that HIPS do not admit the concept of FP ! (They may be right though).

 

As I've already mentioned, it is really hard to test HIPS (both classical and sandbox) defense level. It is really new field, there is no known good methodology of testing (AV/AS and firewalls have it) and tests themself.

 

Ilya Rabinovich as you said

could I ask you how you do test your HIPS product for effectiveness etc ?

Thank you

 

Playing around with the classic HIPS and behvior blockers was fun, but in the end I need to go about my business, get work done, and move on. The Pop-up stuff was interesting for a while but then simply a waste of my time. I've settled on Ilya's DefenseWall (no great suprise as he's over at GSF). Even with DW I found for a couple months I would pour over the event log to see what program was trying to do what to my system... Now I simply clear the log once a day - I don't look (I just don't care).

With DW there are some oddities to be aware of - for instance: I run Opera, if I change the skin, shut Opera down and come back to it, I'm back to the old skin -Hmmmm So to make changes to Opera, even all my email accounts I'll run it once in a while as "trusted" - go over all my settings and shut it down, restart it as "Untrusted". The inconvenience is minimal considering the protection. I can have a young niece or nephew use my computer, browse to their hearts content with no impact on my system "go ahead, beat it up". Truely a joy. At this point I'd dump my AV in a heart beat over losing my HIPS. And I say HIPS because there are many out there that are good, not just DW.

 

Simple- I have a inner set of test examples. Also, I use untrusted registry editor and file manager to test some file system/registry related things. So, it can not be called as a "tests for end-users". To make a real test set for end-users from it- it need many time.

 

Well, Prevx Home/Pro, and to at least some extent Online Armor, were able to track the effectivness of behavior blocking, and the results weren't very good. Honestly, I don't think you're really going to be able to test the real-world effectivness of behavior blockers without doing something similar. Perhaps a scientific type study with a control group and such, which seems a bit much for a technical assesment.

Behavior blockers are made to put all the security into the user's hands. If it takes a malware expert to spot malware with any accuracy (consider that malware is made to fool even advanced users), where does that leave the HIPS user?

 

Well, I'm sorry but this point of view seems narrow to me : Let me repeat that some of the HIPS tested here are able to detect running rootkits as much as preventing them. Does that mean these programs are not HIPS? No. This is detection as you said, but you can see it as the ability to prevent rootkits from hidding as well; this is just a difference of features.

In France we've a phrase for such an attitude as yours : "être plus royaliste que le Roi" (something as "to be more Catholic than the Pope" seems to be the english equivalent ). And I think that IDS are more network-based defences anyway.

I think Kareldjag decided to include some tests with the HIPS disabled to make the tests as wide as possible. 90 % of the tests in part 2 and 3 are based on real malwares, executed locally or from live malware sites, who can say this is not "real-life" tests?

I don't even understand your statement here, and I can use your own words to contradict it :

Sorry, I do not want to be rude here, but just want to show that it doesn't make a lot of sense to criticise such a test. Moore posted a screenshot about this particular situation, let me post another one where you can see what can do a hijacker once it was able to hijack . Obviously there no need for the malware to be installed before the HIPS !!! Why on Earth would it be?



Btw Thank you AJohn, that's a relief to know that at least some people enjoyed these tests .

@ Moore : Your tests in the link you provided are impressive, that's an excellent work. (Can I ask you which program did you use to take this screenshot of PG's ballon tip? Mine doesn't work for this).


nicM

 

I found your tests quite valuable. Please get busy and test some of the other *leading* HIPS -- specifically SSM, OnlineArmor, GSS, & Prevx.

 

I'm not disputing that many HIPS programs can detect a rootkit. My point is that a HIPS program primary task is to prevent the infection or intrusion by not allowing the process in the first place, which includes the installers that ran while the HIPS apps were disabled.
Regarding the "contradiction", I mentioned both legitimate software and malicious code in that statement you quoted.

I did use both app and code, assuming it would be understood that app referred to legitimate software, and "what it's allowed to do" was referring to legitimate software.
I'm not trying to be rude either, but I don't see how some of those tests have anything to do with real life situations. Who besides a tester would shut off their security apps and install malware? Who would install a HIPS application, then disable it and leave their PC unattended? Most malware doesn't get installed locally by someone at the keyboard. It comes from the web as a malicious page, an infected e-mail, a bundled installer, etc. I'm not trying to be picky here and I don't have any problems with the tests themselves. All I'm doing is pointing out that these tests don't reflect reality on several points. It doesn't take a controlled test to figure out that a security app might not protect the user if he disables it. There are way too many things that were allowed in those tests that couldn't have happened if the HIPS software was enabled and properly configured. The rootkit installers wouldn't have run. If the HIPS password option was enabled, a normal user couldn't have disabled the HIPS software. The process termination utilities that use methods that can kill some HIPS wouldn't have been allowed to start in the first place. The user could be denied access to the command prompt with an application rule. Apps that can't run can't install drivers, edit files, terminate processes or set hooks. All this and more could not have happened if the HIPS had been enabled and configured. You can't realistically test how well HIPS defends a system by allowing everything it would have stopped. How can you call that realistic? Other than having a hacker at your keyboard, the only way some of this could have happened is if the malware or rootkit was there before the HIPS was installed.
I'm sorry, but if it's unreasonable to ask that a security application be enabled and both configured and used properly in a test of its effectiveness, then I'm unreasonable.
Rick