Using leak tests to evaluate firewall effectiveness (2007 article)

Exploits targeting specific applications is an attack surface issue. A firewall or security suite is not a substitute for hardening the attack surface, but they can play a role in securing it. For all practical purposes, all attack surface apps are exploitable. The users security policy should acknowledge this by taking whatever steps are necessary and possible to isolate those apps from the rest of the system. Application sandboxes, virtual systems, policy sandboxes, classic HIPS controlling parent-child settings and memory access, reduced permissions, etc alone or in combination are ways to accomplish this. You can't make it perfect but you can make it a royal pain to penetrate, especially if an adversary doesn't know exactly what you set up.

 

People still have this silly notion that firewall with outbound control can prevent malware from phoning home and leak user data. We have had this type of discussion countless of times and yet some people still don't get it.

Apps run on the same desktop can interact with one another. Simple example: Your browser needs to connect so that you can browse the web. What stops malware from hijacking and abusing that?

Here's the thing. There is no proper isolation of desktop apps. Classical HIPS try to give users control over application behavior after execution. Problem is there are just too many areas to monitor. Compare the leak tests in its early days vs what they are tod ay. Leak tests try to measure effectiveness of HIPS but in doing so, acknowledges the fact that there is a lot that can be done once something is allowed to execute. Worse still when it has admin privileges...game over. Even sandboxes on Windows need to make some concessions for apps to run properly.

Betting on firewall with HIPS
to control outbound access of malware is betting on malwarw authors being stupid. That is just too risky. Naive I might say. What you want is to prevent malware(inc. memory-only malware) from executing in its early stages as much as possible. If there is data you cannot risk leaking to the outside world, keep it encrypted and if possible on an air gap machine.

 

That's my thought too. Right now I don't use outbound control, but I was thinking about what's the "next best" thing I could do, without resorting to monitoring everything myself. Are there any free programs firewalls/HIPS that intelligently alert when they see patterns that are suspicious? I suppose Privatefirewall is one that might.

 

A flawed analogy for several reasons. You start with the assumption that malware is already on the system. You state that firewalls and HIPS can't implement a strong enough containment policy to completely restrain that malware. HIPS stands for Host Intrusion Prevention System, not Host Intrusion Containment System. HIPS are very capable of preventing that malware from ever getting on a system, excluding social engineering or user stupidity. They're not the correct tools for implementing a security policy based on containment. Software firewalls (not security suites) are designed to control internet traffic for individual applications and system components. They don't control inter-process activity. Again, wrong tool for the job.

Very true. Some people don't understand the concept of choosing the security policy that's best suited to them, then picking the apps that are best suited for enforcing that policy, not the other way around. Some throw HIPS and firewalls at the problem without bothering to learn their limitations or even how to use them properly, then complain that they don't work. There's a big difference between a pile of security apps and layered security.

Have you ever looked at the system calls an app like SSM hooks? On an XP-SP2 unit, I counted 281. That's the free version of SSM. The pro version covers much more. That said, the term "monitor" is misleading. Intrusion Detection Systems monitor activity, looking for malicious patterns. An app like SSM intercepts that traffic, then either allows or blocks it based on the ruleset. Like internet firewalls, they're only as good as the rules they're enforcing.

 

@noone_particular

I'm very well aware of what HIPS stand for, thank you. I don't deny that there is value in HIPS ability to prevent malware infection, especially if one restricts the main threat gates and set up a default-deny policy to prevent arbitrary execution. We share the same policy but use different tools. You use SSM (a classical HIPS) while I utilize what is built-in on modern OSes like Integrity Levels and browser "sandboxes" like the ones in Chrome which define what a program is allowed to do.

I'm not the one making adsumptions here and my analogy is far from being flawed. You misread my statement. What I'm saying is that there are people who wrongly assume that they can control malware behavior AFTER it executes. Regardless of whether the firewall has an integrated HIPS or not (be it Kerio or Comodo), one simply cannot rely on outbound control to prevent data leakage and malware from phoning home once it has already executed. That is my point...because the thread subject is about leak tests. You and me share the same views but you are too quick to call me wrong while disregarding the context of my post.

As for "monitor", nope it's not misleading. Classical HIPS and HIDS (inc. current AVs with heuristics) are pretty similar in the hooking they do. The major difference is the degree of control. 1 requires you to make the decisions and ruleset while the bother does the analysis and prompts once it finds an anomaly. 1 requires a higher level of system expertise while the other is more automated by utilizing artificial intelligence. There are pros and cons to each but that is another subject beaten to death and yet resurrect from time to time. Another complicated debate.

 

I'm a bit at a loss as to why you view this as "monitoring everything myself". Firewall rules are basically "set and forget". They only need updating when your system changes. When those changes are application updates, all that's usually required is updating the apps signature or file hash in the firewall rules. Beyond that, the only time you should see a firewall prompt is if the app does something the previous version didn't do. In most cases, I'd consider that behavior desirable.

Regarding HIPS, what do you define as "intelligently" alert? A firewall will alert to connection attempts that aren't permitted or blocked by existing rules. Example. A an app attempts a loopback connection to the system. If that app is a browser, it needs that connection to access the user profile. If that app is malicious, the same type of connection could be used to inject its traffic into a legitimate app such as a browser. A HIPS does much the same with applications/processes and their activities. A keyboard or mouse hook can be used by a game to function. The same hooks can be used by a keylogger. In themselves, connections and hooks are normal system functions and activities. Many system functions can be legitimate or be used maliciously. A firewall or HIPS can alert to their usage and/or existence. Additional criteria is needed for the firewall/HIPS to determine if they're necessary or malicious. The HIPS or firewall either gets that information from the vendor or from the user. Options include access to a database with info on these apps, an option to trust signed files, etc. Relying on a database brings it back to the same problems that AVs have, never complete, never completely up to date. Signed files are no guarantee either, especially if governments, defense contractors, or big corporations are involved. The other option is letting the user decide. When apps like SSM and Process Guard were conceived (before the term HIPS existed) it was decided that they would avoid the problems facing most AVs, reliance on signatures, databases, vendor servers, etc. Instead of keeping tract of everything that they should block, they let the user choose what to allow and block everything else. Either way it's a tradeoff.

IMO, the problem isn't the HIPS themselves. The problem is that it's bundled with all kinds of other security apps, then offered to users who have no idea what to do with them. The apps they're bundled with, AVs for example, are based on entirely different security policies. The vendors are trying to make HIPS into something different than they're designed to be and ending up with compromised products as a result.

 

What you are looking for is a subset of HIPS called behavior blocker. Pretty much incorporated and outdone by behind-the-scene technologies that AV utilize today (heuristics, emulation, etc). You are good with searching for info...there are a couple of interesting reads (PDF) on the subject.

 

@noone_particular

The AV vendors are targeting the average person...the end users who do not have the skills or inclination to use classical HIPS. So, designating AI technologies is while imperfect, more fruitful/reliable than to leave the decision-making to them. They aim to make the distiction between good vs bad...as compared to HIPS. You know...PEBKAC. Not everyone can employ HIPS effectively so while you may say you on't agree with the 'dumbing down', it has the potential to help these users than without anything. I'm not an AV fan myself but they have a role to play. I'm sure you will disagree so I will leave it at there. Just wanted to voice my opinion.

 

This does not make sense of course, or only from a marketing point of view...
Just tell me how these tool can evaluate the packet filtering effectiveness of these firewalls
https://www.wilderssecurity.com/showthread.php?t=303141
Or the effectiveness of an hardware firewall like a Bintec one that i have used 2 years ago...
Gkweb used leaktests to evaluate desktop firewall as a personal methodology, i used leaktests to evaluate HIPS as a personal methodology, and other in France, Germany, and Russia too.
Matousec has made this methodology much more popular for profits.
But in all case these tools are not the right way to evaluate soft or hard firewall mostly designed to deal with OSI protocols.
In fact, marketing departments often change the destiny of products, have seen the benefit of tests scores and stars, and we have seen the raise of desktop firewall that integrate HIPS module, and in some case AV scanner, and finally to become not av suites, but Antimalwares in general.

I exchanged some mails with N.Grebennikov in 2007, and as honest an av developer is, we always need to remember for which kind of editor he works...
Before him, another article in French provides an overview of methods used to bypass firewalls, article well known by Look'nStop users
http://www.tdeig.ch/windows/contournement_pfw.pdf

Rgds

 

@noone_particular: I view the Matousec tests as both host intrusion prevention tests and also leak/spy tests. Like safeguy, I view my host intrusion prevention measures as things like standard user account, UAC, tight permissions, EMET, anti-executable, integrity levels, etc. When I used XP, I used full HIPS like SSM and Comodo; I don't want to do it again. Using just basic firewall rules (e.g. "Program X is allowed to connect to the internet") will stop some malware/greyware from leaking, but not some others. To do a better job against leaking requires HIPS though (do you agree?), and as I said, I don't want to do it again. So what's the next best thing I could do? I'd like a program to do behavioral analysis instead of me doing it. Some might call that an expert system; some might call it a behavior blocker. As safeguy notes, AV often does this sort of thing nowadays; I'm wondering if there are any free HIPS/firewalls that can also do it.

 

safeguy,
I did misinterpret your post. Sry. I don't trust any sandbox to contain malware once it executes. On an NT system, I like Sandboxie for its ability to eliminate usage records. As a primary defense, I consider sandboxing insufficient, whether it's freestanding or part of an app.

Regarding the built in security features in the current versions of Windows, I have no trust in them. IMO, ASLR is of very limited value if it isn't applied to all the processes. Their calling the LoadLibraryEx bypass a feature sealed it for me. I have to wonder how many more of these "features" are waiting to be discovered.

MrBrian,
We'll have to agree to disagree on Matousec. AFAIC, his "testing" is little more than extortion.

Regarding vendors targeting the average user, as long as unskilled users can function as administrators, nothing is going to change in the long term. No amount of integrity levels, HIPS, or the rest of their alphabet soup security features will make any real difference.

 

I'm not sure that we disagree. I had some harsh words for the level methodology used. Whether the tests themselves are valuable for either host intrusion prevention testing and/or spy/leak testing is a separate matter, IMHO.

 

IMO, using them to test or "rate" the apps themselves is fundamentally and deliberately flawed. How well any individual app performs against a test that supposedly simulates malware is of very little consequence. The only thing that really matters is how the entire package performs as a unit.

Using leaktests for evaluating or tightening a configuration is sensible, but if one has to violate their security policy in order to run them, what is really being tested? The users willingness to "click here" when they should know better? If one is relying on a security policy based on containment, leaktests have some value in determining if their containment is effective against the commonly known methods. For evaluating a default-deny policy, they're worthless. Even with AV based default-permit policies, they're of almost no value. In order to make leaktests realistic, the attack simulation should come through the normal attack surface, through the browser, contained in a PDF, etc, not "download this and click here".

In one form or another, this battle has been being fought for at least the last 10 years with application and system updates against privilege escalation and remote code execution attacks. The only thing that's really changed is that sandoxes are now recognized as separate apps or components that are part of other applications. Instead of attacking an app directly, the sandbox becomes the target. Like so many other "innovations", they'll enjoy success for a while, until the attackers figure them out. After that, we'll be right back where we started, again, with one minor change. Instead of penetrate, patch, and repeat, it's escape, patch, and repeat.

Regarding the including of HIPS to security packages that target the average user, what is that user going to do with it other than rely on its "out of the box" settings? Is there even a point to giving them an interface for it? On their weaker, more user friendly settings, most HIPS will fail many of those leaktests. Using settings that will pass those tests, HIPS will interfere too much with normal operations unless the user makes specific allowances. What will determine that a keyboard hook is okay for one app but malicious if used by another? A vendor database like those used by AVs? They can't maintain a database of everything that's malicious. How in the world would they maintain one of everything that's not? The privacy implications of such an arrangement are insane. Such a system would have to call home every time you install or update an app.

I don't see a long term solution that's effective for the average user using Windows in its current form. As long as Windows is default-permit and allows the average user to play administrator, we're running in circles.

 

@noone_particular: I like whole-product tests. However, when testing a certain part of a security product, IMHO it should be tested in isolation of the other parts as much as possible, because one wants to know how good that part is.

 

The problem I have with testing a product in isolation is that such a test favors security packages. How do you test one component of a security suite without the rest of the components affecting the test results? Will such a package allow you to shut off the firewall and AV components in order to test just the HIPS configuration?

 

Why is there a discussion about a 2007 test methodology ,whats the purpose of this thread ?!
We are in 2014.

 

Exactly.

Exactly.

Exactly.

Exactly.

Yet few things have never changed

The most missing part of every sec. suite test is the human factor. With a proper social engineering users click on anything. So all the tests (meaning testing businesses outputs) are more marketing papers than a reflection of reality.

 

1. That link had been posted only once at Wilders before if I recall correctly, and that was a recent post buried in some other thread.
2. "2007" is in the thread title, so there was no attempt to mislead anyone.
3. I think it's still useful today. Some may disagree.
4. Got a better, more recent link covering the same type of material?

 

@noone_particular: I think you and I don't have much of a difference here:
1. We both try to prevent host intrusion, albeit using different methods.
2. Neither one of us focuses much on leak prevention. You use basic outbound control. I use none currently, but am thinking of changing that.

 

Most of this has not changed. Internet address structure, ports, basic protocols, the DNS system are basically unchanged for at least 20 years. Internet firewalls work basically the same as they have all along. A lot has changed in operating systems, but much more has not. Software and operating system vendors are quick to point out new things, new forms of built in security (ASLR, DEP, IYL), new features, etc. They're very quiet about how much remains the same, which includes the way the majority of things work. Except for HTML-5 and some more recent javascript, Win 95 and 98 still run on the web much as they always have.

The primary difference I see is our choices of what we trust. Microsoft lost my trust completely with Vista, 7, and 8. Ports that can't be closed. Security features bypassed by design with the likes of LoadLibraryEx. File systems that can hide executables from the user in ADS. Multiple logs and registry entries containing everything that you viewed, installed, used, etc. I realize that everyone assigns a different weight or level of importance to these and many other features/liabilities of an OS. For me, the shortcomings and liabilities outweigh the benefits.

 

From IAmA a malware coder and botnet operator, AMA:

 

Why should an internet firewall be expected to compensate for the rest of the security package (or the user) that failed to protect the system in the first place? In the scenario above, why wouldn't the firewall be evaded? Everything else has. If there's an AV, it failed to detect the malware. If there's HIPS, it failed to stop the injection. If there's a sandbox, it allowed the malware access to the stored passwords. The user failed on all counts. A firewall is one tool, not a cure-all.

 

Do you view outbound protection as useful in practice for 1) preventing host intrusion, 2) for detecting/preventing data exfiltration after host intrusion, or 3) both 1 and 2?

 

Outbound "protection" is overrated. There you go, I've said it.

Against malware, it's useless. Against software trying to "phone home", you still need to trust that the software would not try to piggyback on connections that are allowed. With legit clean trustworthy software, that is less an issue (hopefully)but against everything else, no.

Simple fact: A software firewall with or without HIPS cannot reliably prevent leakage (as leaktests have proven over time). Only some people refuse to see the truth.

You use software firewall to control/restrict outbound connections of software that you already trust. For e.g, you might want to restrict your browser to ports 80 and 443 only and/or your other apps to connect to certain IP range. You don't use software firewall to stop spyware/keyloggers from sending your data back to its master.

 

From paper "NetGator: Malware Detection Using Program Interactive Challenges" (2012):

HTTP ~77%. HTTPS ~3%. Other (SMTP, FTP, ICMP, IRC, DNS, etc.) ~20%.