Personal HIPS Tests

No, you cannot make direct conclusions from this.

No. If we're talking about time in seconds, than the importance of this is overstated.

Rather, (and I'm not sure if richf already said this) it is the amount and type of 'work' done before detection that is critical.

Early detection of a 'testing app' making a type of function call which is commonly used for legitimate purposes smells suspiciously like a signature-based detection mechanism for this app rather than a generic detection ability. (I've said in the past that this may be going on with the detection of certain firewall leaktests by firewalls). On the other hand detection of malicious behaviour just before a shutdown may be far too long.
Installation order may affect detection time, but startup order may also be important. It is somewhat tricky if not difficult to load a large selection of apps in a certain order on startup, especially when some have both a application-service and driver that must be loaded for full functionality. Additionallly, this is no guarantee that one or the other will take priority in cpu or memory usage. Hence, it is difficult to make conclusions from this kind of data by itself. As much as what vendors might say otherwise, this really is conflict territory, although it's due partly to good programming and the rarity of certain user behaviour that there it isn't more software conflicts.

 

Yes, this is what I was suggesting. It is not "time" but "type and amount of work performed", that is critical. If the malware is stopped from doing any work, including instantiating itself in the operating system, then it has been effectively stopped from doing harm and prevented from doing any harm in the future.

Best,
Rich

 

Really? That's AV territory? Anyhow I manage to rule this out with a certain degree of certainity .

AV catches it first, on running.

Impressive. Though surprising to hear. I believe you stated that you have started several companies has well in the tech sector as well ?

Personally I think Diamond should pay you to be a consultant, if they haven't already.

Well this doesn't seem very productive, if a software wants to cheat it would be much better to key on something more specific in the 'testing app', say it's file hash.

If HIPS keys on a function call that is legitimate wouldn't that cause unncessary prompts for other software?

Ghost, ever though of testing HIPS? It a little related to leak tests, though a whole lot more complicated.

 

How to test HIPS

I think this is a very tricky subject.

Unlike say antivirus tests where products from different vendors have roughly the same functionality - detect viruses*, HIPS programs each check a unique and different pool of behaviors.

There is some overlap but as the industry for HIPS is still young, people haven't yet gotten sufficient time to come to a unspoken consensus on what is most cost effective in terms of resource use, user friendliness and pure irriation level to monitor.

So we have a big pool of HIPS each with some common features but with even bigger pool of different feature sets. Innovation at it's peak!

As such In theory, educated users would also want to know a little about what exactly is covered and what isn't. The default assumption without any other evidence to the contratry I guess would be if HIPS X was covering a very restricted set, and Y had a broader set, Y might be worth looking at, all else between equal.

The problem for users and testers are this.

For the tester

How do we go about testing HIPS objectively?

One approach I see used by some (less technical) reviewers is to unleash as many real world threats as they can, and see how many are detected/stopped.

Pros : Nothing beats real world tests. Test that claim to isolate singluar effects (e.g. changing registry keys, writting physical memory e.t.c) to see if the HIPS reacts are unrealistic and useless to the average man in the street.

Cons: Difficult to interpret. Barring a 100% indentification obtained from a blacklist, how do you decide if the HIPS as really stopped the malware? Does it pass only if it automatically removes part of it? Does it pass if it merely tells you an unwhite listed installtion program you started a second ago is trying to start?


Another approach is to try to run proof of concept (PoC) tests
that demostrate a certain vector of attack and to see if the HIPS programs catches them.

This usually but not always corresponds to some behavior that HIPS are designed to catch. The well known dll injection is probably a famous one.

This is what Kareldjag tries to do, most of the time.

Pros : Easier (in a sense) to interprete with the right test. Either you detect the hook and block it, or you don't. Either you prevent the termination of a program or you don't.

Allows you to test claims of vendors. You say it blocks buffer overflows? Well this test says it doesn't block stack based Buffer overuns.

Results of test, allows you to guess how well or how much of a real world malware will be stopped, given a well analysed malware.

Cons : Unless the tester has written the PoC tools themselves, it's hard to be sure what they are doing 100%. As pointed out by ghost16825 it's unclear what some tests such as TrojanDemo are doing.

Interpreting the test is not very clear cut either

Slightly more technical to understand - 'My HIPS fails the KAIPMON test, what the heck does that mean??'

Failure to block a certain vector doesn't mean the HIPS is bad, because it might have compensated in another area.

Vikorr points out a related issue. Is it better for a HIPS to provide weak but inferior protection in a given area, than to state up front it provides none at all?

A HIPS using polling methods doesn't fully past certain registry insertion tests, does it make it better ot worse than one that does not provide any registry protection at all?

Lack of information/ability to design special tailored tests for vendors claims . Witness 'Our super duper HIPS provides zero day protection against all classes of worms, spyware and trojans. All without the use of signatures!' . Er.. Barring further technical details on how it does that, how do we actually test that?

For Users, the challenge is to decide what to use. It's bewildering really, there is no objective standard out there to appeal too.

As contentious as virus test results are, people do get a feel about the detection rates of antiviruses.

In the field of HIPS, I can't see how even that can be achieved.

Thanks to kareldjag's groundbreaking and sterling work, a dilgent reader can with some work now answer questions himself about whether Safe N Sec overlaps 100% with Processguard, or whether he should keep Regdefend with Online Armor (once it's reviewed) by studying Kareldjag's blog.

But that still doesn't help one decide if failing a specific test is critical or not. Or do we choose a combo of HIPS so that they combine to pass all tests

PS Not that picking AVs are easy, but HIPS are a nightmare.

PSS How many HIPS should I have. Everyone seems to be running at least 2, some even 4.




*Even then, there is dispute over method an antivirus should detect trojans, spyware etc....
** And in this depends on circumstances

 

Re: How to test HIPS

Very informative analysis.

thanks,

-rich
________________
~~Be ALERT!!! ~~

 

Re: How to test HIPS

Yes, indeed.....I learn something new everyday

 

There are two primary goals of HIPS:

1) To stop executables dead in their tracks. This would be possible if all points of entry are known, and all ways of stopping executables at these points of entry were known. This is the first line of defense.

Unfortunately, Windows XP is too complex to have 100% assurance of this. However, because HIPS vendors are concentrating on "points of entry" rather than "what is trying to enter" (numerically there are far, far fewer "points of entry" than there are potential "bad guys" who are trying to enter) they can stop all that is known, and it is easier to adjust as new vulnerability points are discovered. However, in order to evaluate a product, the user needs to know all currently known points of entry and whether their HIPS program is guardeding these points adequately. At this point the user is unfortunately guessing but kareldjag's tests shed lots of light.

2) To keep malware from instantiating itself. This is the second line of defense. If malware cannot instantiate itself (e.g. change registry entrties, autorun files, etc. , it cannot cause repetitive harm. This is a good backup for the case where malware does get to execute (through a new, unknown vulnerability point) and begin to cause repetitive harm through re-boots. Products such as DeepFreeze and ShadowUser take a very absolute view of this while products such as RegDefend, Prevx are more relaxed. Both approaches have their strengths and weaknesses. This "second line of defense", only exists because for some reason the executable got past the first line of defense.

Beyond this, users can still use classical signature/heuristic detection to help identify malware from gaining entry into the system (AVs) or killing them once they have gained entry and are processing (ATs). Both of these types of programs are only necessary because HIPS are new and still unproven. The new ZoneAlarm Pro program contains a bit of the classical approach and newer HIPS and it should be interesting to see how users perceive this new product since the ZoneAlarm population is very large and diverse.

Tests can be categorized within these two primary goals, and it would give users a good idea of how well they are covered by known first line of defense (stop executables) and second line defense (stop instantiation).

Rich

 

Not on my system.

Yes, I have had a very long and varied career.

Rich

 

Very comprehensive, thorough analysis, Rich.

Should the first line of defense depend on a product, or one's own decision as to what is installed on the computer?

-rich
________________
~~Be ALERT!!! ~~

 

Hi Rich,

Thank you. I very much respect your knowledge and you have been very helpful to me in the past. So I feel very complimented by your kind remark.

In my view, what is most desirable from a vendor's, consultant's, and customer/users perspective is:

1) An independent list of known vulnerabilites with respect to the "first line of defense". This is most important, since once a "bad guy" gets in, the number of vulnerable points as well as the potential damage, is innumerable.

2) Once such a list has been enumerated, it would be nice to have a set of possible solutions to close each one of these vulnerable points. The solutions can be classified under such headings as: 1) User behavior 2) Operating system adjustments (i.e. "tweaking), 3) software solutions. etc. Each one of these solutions will have its pros/cons: e.g. absolute protection may interfere with user workflow, or the solution may require user judgement instead of software judgement, etc.

It would be a good project for researchers to create such a list and begin to fill in the solution list. I do not think this would be overwhelming. Personally, I feel that the work you did and shared with me on understanding and controlling script behavior should be in the public domain, so that this aspect of vulnerability control can be discussed. I think it is important to analyze this aspect of "first line of defense" control.

What may require much work, is testing who well each solution actually works in real-life. So of course, there will always be differences in opinions on this count - and certainly plenty of room for competition.

Any comments are more than welcome.

Regards,
Rich

 

Exactly my point.

 

This is referring to the first line of defense, one that I've thought about a lot.

DiamondCS' wonderfully documented "attacks" site:

http://diamondcs.com.au/processguard/index.php?page=attacks

shows how Process Guard works at the second line of defense - "To keep malware from instantiating itself" - to use your phrase.

Take their rootkit test: a rootkit is a trojan, which requires first, that the trojan, or "dropper" become installed, - gets past the first line of defense - and second, that the dropper load a driver or dll - instantiates itself. PG (and others) prevents the instantiation.

All of these test are possible to run only because the user has permitted the malware to come into the system so as to run the test. I have run most of the DiamondCS tests myself. You state it this way:

------------------------
This "second line of defense", only exists because for some reason the executable got past the first line of defense.
------------------------

So, the question is, in our real, not testing world, how to keep the executable from getting past the first line of defense.

Taking the rootkit as an example (it seems to be quite feared today) - - you can ask yourself, "how could this get installed? I would have to be tricked, because I wouldn't knowingly install it. What is the probability that this would happen?"

At this point, assuming that products/programs you purchase from reputable vendors are safe, one can conclude that a rootkit can enter only via something downloaded from a not so reputable site, or from a non-trusted source, I would volunteer to say that your first line of defense can depend solely on your own judgement/knowledge, and your second line of defense (however you want to set it up) can take care of the remote possibility that it did happen.

regards,

-rich
________________
~~Be ALERT!!! ~~

 

It all depends upon the signatures. Some AV companies may not want to warn against simulators.

 

I totally agree with your overall analysis.

I do think that there may be ways for vendors to augment user judgement with "advice" from their own software analysis and database. In a sense, this becomes a new form of "signatures" or "heuristics". Some vendors may choose to make decisions entirely on their own in the software package (removing some decision making from the user), others may simply "alert" and leave it up to the users.

HIPS, as far as I can tell, tries to alert users that something unusal is "knocking at the door". A user still needs some way of making a decision, with the help of a "database" (could be personal) or some "smart knowledge" based upon experiences (heuristics). But at least the door is now being watched. The problem that I always had, is that I knew that the door was open, but couldn't figure out how to close it and watch who I let in. This was why I was reasonaly excited when I first came upon ProcessGuard, WormGuard, and RegDefend. Developments by other companies entering into this arena are very interesting to watch.

Regards,
Rich

 

Re: How to test HIPS

Yes me too. Posting rubbish seems to be sufficient to impress you.

Some more rubbish then.

First the trival.


HIPS = Host based intrustion Prevent system. The word 'Host' I gather refers to the fact that the defensive software runs on each 'host' workstations in the corporate environment as opposed to centralised firewalls/Antiviruses on mailservers on specialised checkpoints/servers.

From the home user point of view, isn't everything used almost like a HIPS?

Why does HIPS seem to almost always imply that its not using classical signatures? Is it due to the letters IPS ? Maybe due to historical reasons stemming from the acronym IDS?

The Term HIPS I think is not very informative.

Clearly disagree with this. 'Classical Signature' (What's a none classical signature?) based approaches are always the prefered way to handle malware, because it's more specific.

So it will always be necessary.

By stopping executables dead in their tracks, what exactly do you mean? Do you mean stopping processes from starting that are not directly (whatever that means) user initated or even a whitelist?

Altough their seems to be an unending number of exploits that allow 'remote execution of arbitary code', which enables (this is where the cleverness lies btw) processes to be run without user permission. However in practically all cases the starting processes will be caught by simple hooking.

Many AVs and ATs already do this since the beginning, the problem as always is not spotting them when they start, but being able to spot a evildoer.

The only different between classical AVs and HIPS is that the former only stop the process if they can tell via signatures if such a process is a baddie while the latter, uses other methods.

Of course, some HIPS solution do execution launch protection .They use the behavior 'process starting not on white list' as a flag/signature for possible malware.

A very powerful feature no doubt, but one that has a high rate of False positives (assuming Antivirus terminology is applicable) and practically useless against traditional trojans. I don't see this catching on for various reasons.

Neither do I see it as a major function of a HIPS. It's just one of many behaviors that can be flagged.

Since we arent allowed to use classical signatures and we don't have magic we have to judge on behavior. And this means allowing the process to do a bit of 'instantiating'.

This is where, HIPS can get very very clever and I think is the heart of the HIPS engine.

What behaviors (or combinations) should they tag? Should I go for behaviors that might be used by malicious processes even though they can be legimate? What if after 'instantiating'
the damage cannot be reversed?

I can think of several different answers.

 

Excellent. Exactly my thoughts.

 

Yours does. Not that it matters to my point one bit.

 

I hesitate to interprete for Rmus, but what he is saying I think is that your 'first line of defense' is not the main function of a HIPS.

The key word is 'unusual'. In practically all cases a process starting would not be unusual.

 

Re: How to test HIPS

I get zero "false positives" from either ProcessGuard or RegDefend. Every alert I have received was accurate. As to the rate of alerts: On my system it has quiesced to almost negligible. Some people are party animals (high risk internet behavior) and they will get lots of alerts. Others have a more quiet "homelife" (i.e. computer access behavior) and I don't have many "new people' knocking on my door, so I have very little to check. I think the vast majority of people, will find it quite comfortable learning to handle new, "uninvited visitors", once their HIPS product is in place. It took about two days for Online Armor to find its happy quiescent level on my system. Not bad at all. And this is a brand new product!

Not at all. ZoneAlarm Pro utilizes both approaches: Anti-spyware signatures, for example, as well as OSFirewall/SmartDefense techniques (e.g. executable program whitelists). Seems like a very reasonable approach to me.

 

Of course, though I agree, I think you might be slightly overstating matters. Exploits in browsers, emails etc have existed that cause automatic execution of code.

That is where Richrf's first line of defense comes in.

The problem I think with this though, is that it's far too costly in terms of effort to whitelist every process.

If there was only some reliable way to tell the difference between a process started directly by a user initated process and one that isn't, you would have a very powerful behavior to flag. You wouldnt have to waste time, clicking yes, to a process you started one second ago, and yet you can catch processes starting you didnt directly start.

In some ways this has already being done, eg exes starting from temp folders are flagged as dangerous by some HIPS

In practice though, this is possible only for a small subset of actions and only via indirect means.

 

I'm not the one to ask about this - I don't understand how it works, and the various tests and conclusions are confusing to me. Maybe kareldjag will shed some light here. He has certainly looked into this in depth.

-rich
________________
~~Be ALERT!!! ~~

 

In fact, it may turn out that this is extremely inexpensive for the vast number of PCS out there. It may turn out, that simple lockdown programs, e.g. Anti-Executable coupled with DeepFreeze, which can reject all new executables (as well as operating system updates), except for a handful of update modules, is all that the average user needs. The problem may not be in the "problem" but the way vendors have perceived the problem. In fact, vendors may be building products directed toward "high risk" users, while ignoring simple solutions for the average user who is just doing email and maybe some financial work online.

The solution to most customers/users in this world may actually be quite straightforward. Remains to be seen.

 

Re: How to test HIPS

Let's no quibble about what false positives means. The term as I said probably isn't applicable.

Though the user effort to handle the prompts is often underestimated.

This brings me to a point that is puzzling me.

Party animals I suppose might refer to people who keep installing new programs. A lot of people here enjoy testing new applications so I would expect them to have a lot more prompts than the typical user due to new processes.

Yet people seem to maintain that they get few prompts. I wonder why. Perception is a funny thing I guess.

I think the best way I think to settle this is to do a proper quantification. Actually keep a 100% record of the number of prompts you get. I was thinking you could even divide them between prompts that were expected (after immediately running something) and unexpected.

Start from the time you installed it until it stablises.

How many prompts daily do you think you will get? People often say 'rare'.....

Anyone interested?

My point was in regards to using behaviorial approaches to detect malware. Something HIPS are synonymous with today.

Of course no one is ruling out a combined approach. Safe N Sec with bitdefender (great combo!!) for example.

I'm a bit touchy when people seem to be talking as if signature based approaches will be dead and useless. BTW

 

I'm pretty sure the reason why such programs are used in the corporate environment is that for them whitelisting all processes is a doable thing.

You are after all supposed to do work, updates to the operating system are centralised etc and hence there is a very limited number of processes that are expected to run.

And if something the user wants to run doesnt? Oh well, it's not his computer anyway so he has nothing to get upset about.

Compare this to a home user, who does not have a well defined limited subset of actions. We often say the average user _only_ emails and browses the web, but I think this is a very very small population. Almost everyone I think runs programs they see on the net, or programs people recommend them.

How do I know? Well just look at how many people get infected )

Add the fact that each home user system will be a mix of different software of different versions, you have to whitelist every version of even well known programs. Can we say resource nightmare?

 

When we talk about HIPS stopping something dead in its track, as compared to something that has already instanted itself, it's easy to get confused.

Where is the dividing line between the 2? Is there one?

Clearly something has started, otherwise there is nothing to stop

Here's where it gets technical.