New tests from Matousec

[Dwarden]

In case someone missed these ...

2008-05-06: Three new tests have been added to the suite. PerfTCP and PerfUDP have been added to Level 1, SockSnif to Level 8.
2008-04-24: Seven new tests, namely Keylog1, Keylog2, Keylog3, Keylog4, Keylog5, Keylog6 and Keylog7, have been added.

http://www.matousec.com/

-{ Quote: "
We have implemented three new tests and added them to Security Software Testing Suite (SSTS) and also to Firewall Challenge (FWC). There are two new performance tests, PerfTCP and PerfUDP, that measure the impacts of using personal firewall on the network performance. The last new test is called SockSnif and it tests the protection against unwanted packet sniffing. Firewall Challenge now contains 73 tests.
" }-

-{ Quote: "
We have implemented seven new keylogger tests to Security Software Testing Suite and together with ShadowHook we have added them to Firewall Challenge. We have 70 tests in the system of Firewall Challenge now. Newly tested products as well as new versions of already tested products are always tested against all the tests in the system on levels that they reach.
" }-

i would like to avoid the 'endless' discussion about Matousec group motives or style so please comment the tests and methology self ...

[subset]

Hi,

a comment on the tests.
Do vendors have to pay another time for the added tests to be tested on already tested programs?
Or will this new tests be tested and added for free to the tests of already tested programs?
Just for curiosity. :dry:

Cheers

[Makav3l1]

I don't understand why he tests pure firewall programs as if they were hips. Or why he tests only the firewall from security suites. Of course they aren't going to do well. This guy needs to call it the hips challenge and stop testing stand alone firewalls.

[LoneWolf]

-{ Quote: "I don't understand why he tests pure firewall programs as if they were hips. Or why he tests only the firewall from security suites. Of course they aren't going to do well. This guy needs to call it the hips challenge and stop testing stand alone firewalls." }-

I could not agree more.

[subset]

Hi,

this may sound dumb, but why is for example the AV modul from Avira Suite forbidden and has to be shut down during the tests and the HIPS module from Online Armor is allowed to run during the tests.

As said before before, this tests are determined as "Firewall Challenge" by the vendor.
But on one side blacklisting the tests for AVs is forbidden and on the other side HIPS are not forced to add the tests to their whitelist.
Or at least AV modules and HIPS modules have to be shut down during the tests for a real "Firewall Challenge".

Cheers

[Coolio10]

-{ Quote: "Hi,

this may sound dumb, but why is for example the AV modul from Avira Suite forbidden and has to be shut down during the tests and the HIPS module from Online Armor is allowed to run during the tests.

As said before before, this tests are determined as "Firewall Challenge" by the vendor.
But on one side blacklisting the tests for AVs is forbidden and on the other side HIPS are not forced to add the tests to their whitelist.
Or at least AV modules and HIPS modules have to be shut down during the tests for a real "Firewall Challenge".

Cheers" }-
Default settings?

[ggf31416]

-{ Quote: "
but why is for example the AV modul from Avira Suite forbidden and has to be shut down during the tests and the HIPS module from Online Armor is allowed to run during the tests.
As said before before, this tests are determined as "Firewall Challenge" by the vendor.
" }-

Firewalls and HIPS will react to both known and unknow malware, unlike AVs which only reacts again known (or detected by heuristic) malware. Also testing with the AV enabled would be meaningless as every vendor would add the tests to the blacklist and get 100% in 5 seconds, without increasing protection against real malware using those techniques.

Nowadays most firewalls are not pure firewalls but have some HIPS components. If you disable the HIPS in a firewall you should disable almost every feature in the rest, because they are also HIPS. Almost every leaktest would fail if you disable these features.

[Dwarden]

-{ Quote: "I don't understand why he tests pure firewall programs as if they were hips. Or why he tests only the firewall from security suites. Of course they aren't going to do well. This guy needs to call it the hips challenge and stop testing stand alone firewalls." }-

that's described in detail on Matousec website and was discussed to death on this and comodo forums x times etc...

[subset]

-{ Quote: "Also testing with the AV enabled would be meaningless as every vendor would add the tests to the blacklist and get 100% in 5 seconds, without increasing protection against real malware using those techniques." }-
Does this mean these tests are meaningful because AVs are forbidden ???
Either they test programs like they are or they test just firewalls only.

Now this are just crude tests, far from reality because of their strange settings and without any value for users.

Reality would perhaps look like this (again for example Avira and OA):
- Avira's heuristic might detect most of the files as HEUR/Malware, TR/Hijacker.Gen or TR/Proxy.Gen etc. >>> user is mostly well-protected.
- Online Armor HIPS might alarm the user with multiple popups about bad things going on. >>> user is mostly well-protected.

So my conclusion is, these tests are pointless because the methology is apparently made between the Ivory Tower and the Temple of Mammon.

Cheers

[ggf31416]

-{ Quote: "
Reality would perhaps look like this (again for example Avira and OA):
- Avira's heuristic might detect most of the files as HEUR/Malware, TR/Hijacker.Gen or TR/Proxy.Gen etc. >>> user is mostly well-protected.
- Online Armor HIPS might alarm the user with multiple popups about bad things going on. >>> user is mostly well-protected.

" }-

1) Unlike a firewall or HIPS, if the AV (through signatures or heuristics) detects the leaktest it doesn't mean that will detect all malware that uses the same method
2) Testing AVs is quite different than testing HIPS or firewalls. Also there are other well-known AV testing organizations.
3) HIPS are much closer to firewalls than AVs to firewalls. An Antivirus analyses the code of the programs and is based mainly on blacklisting. Firewalls and HIPS monitor the behaviour of the program and are based on whitelisting.

[The_1337]

hips isnt a firewall either so i dont see why that would be included. i mean it's called a firewall challenge not firewall and hips challenge.

[Einsturzende]

-{ Quote: "hips isnt a firewall either so i dont see why that would be included. i mean it's called a firewall challenge not firewall and hips challenge." }-

But personal firewall isn't complete if cannot distinguish which process made net request, so some technique similar to HIPS must be implemented (and tested of course)...

[Zombini]

Matousec is a waste of time.. ignore them. The only products that get a 100% percent on those tests are products that are not usable in the real world.

[Dwarden]

-{ Quote: "Matousec is a waste of time.. ignore them. The only products that get a 100% percent on those tests are products that are not usable in the real world." }-

write better tests and share source ... same like Matousec does

[dawgg]

-{ Quote: "Matousec is a waste of time.. ignore them. The only products that get a 100% percent on those tests are products that are not usable in the real world." }-
Uhhhhhh... yes they are usable in the real world

[JanPoko]

-{ Quote: "Matousec is a waste of time.. ignore them. The only products that get a 100% percent on those tests are products that are not usable in the real world." }-

Comodo and Online Armor are not usable in the real world?
Thanks for that info, until now, I was convinced opposite.::)

And using both of them (comodo first, oa now) to my complete satisfaction. And based upon my long term experience calling both of them really the best firewalls available !!!

[subset]

Hi,

kindly note that using Comodo FP or Filseclab PF or whatever does not make Matousec's tests and methology meaningful or meaningless per se.

However, the only noticeable result from this "Firewall(?) Challenge" is strangewise:
You can estimate, that an application has some sort of HIPS(!) features or not.

You can not even estimate, that an application has a Firewall included (ProSecurity...).

Blank refusal, try again, fail better.

Cheers

[Pedro]

I think you'll find that most people, regardless of their opinion on Matousec, are tired of discussing the whole HIPS or FW, or whatever.

Now he comes with some tests that should interest everyone:
-{ Quote: "There are two new performance tests, PerfTCP and PerfUDP, that measure the impacts of using personal firewall on the network performance. The last new test is called SockSnif and it tests the protection against unwanted packet sniffing." }-
So instead of pushing the same arguments back and forth, we could try these?

Cheers

[Einsturzende]

-{ Quote: "
...You can not even estimate, that an application has a Firewall included (ProSecurity...)..." }-
I think "Accessing Network" thingy in it is good enough for testing it like an "firewall" and with tests arsenal Matousec uses for testing FWs.

[Zombini]

-{ Quote: "Uhhhhhh... yes they are usable in the real world" }-

Maaaaybee by a small fraction of computer users. I can bet most people even on this newsgroup dont even know what an IP address is, so even the simplest alerts that say "IP address" are totally meaningless to 99.999% of users. Comodo and others ofcourse take the meaningless alerts to a whole new level.

[Einsturzende]

-{ Quote: "Maaaaybee by a small fraction of computer users. I can bet most people even on this newsgroup dont even know what an IP address is, so even the simplest alerts that say "IP address" are totally meaningless to 99.999% of users. Comodo and others ofcourse take the meaningless alerts to a whole new level." }-
On the contrary...

[Diver]

SockSniff, sounds nasty.

I have always felt the whole concept of detecting malware by waiting for it to phone home is nonsense.

What is needed are ways to block malware installation in the first place, or at least detect its presence without relying on signatures. Products like Threatfire are a step in the right direction, but are probably not all that reliable yet.

[dawgg]

-{ Quote: "Maaaaybee by a small fraction of computer users. I can bet most people even on this newsgroup dont even know what an IP address is, so even the simplest alerts that say "IP address" are totally meaningless to 99.999% of users. Comodo and others ofcourse take the meaningless alerts to a whole new level." }-
99.99999& of users is a over exaggeration. Yes, a vast numbers of users don't know about IP addresses, but the firewalls are still usable by people. Even if they do not know what an IP address is, they can still use it by using an Allow All option rather than just allowing the IP address.

I wouldn't say the software which scores well on leaktests are "not usable".

-{ Quote: "I have always felt the whole concept of detecting malware by waiting for it to phone home is nonsense.

What is needed are ways to block malware installation in the first place, or at least detect its presence without relying on signatures." }-
The thing is, at the end of the day, with the present technology, its still possible for malware to be installed in the first place and leak data or automatically download data, and a leak proof firewall is a method to plug this hole for many users.

[Bunkhouse Buck]

-{ Quote: "SockSniff, sounds nasty.

I have always felt the whole concept of detecting malware by waiting for it to phone home is nonsense.

What is needed are ways to block malware installation in the first place, or at least detect its presence without relying on signatures. Products like Threatfire are a step in the right direction, but are probably not all that reliable yet." }-

It is nonsense. Once they are in your house, they can figure out a way to steal your stuff. The entire anti-leak concept is fallacious in my view.

[Mrkvonic]

Hello,

Like I've said a kabullion times, leaktests only show how easily or in how many different ways the operating system can be fooled. Since the firewall is installed on top of the kernel, it seems like a self-defeat logic to try to control that kernel. After all, the firewall will do only whatever the kernel decides to let it see and process.

Unless the firewall becomes some sort of super service for the kernel, which is absurd. You might as well try a different kernel.

Furthermore, leaktests are ineffective, because:
- People test them deliberately, knowing what to expect. It's easy blocking something called thermite or whatever, but what about something like internet explorer or explorer or svchost.
- They assume you have been infected, which is the worst thing you can possibly do; like drinking poison and then testing if your liver can take it.

Finally, a very good piece of malware will subvert the kernel, change the tcip stack etc - you will get precisely 0 prompts from your firewall.

Mrk

[alex_s]

-{ Quote: "It is nonsense. Once they are in your house, they can figure out a way to steal your stuff. The entire anti-leak concept is fallacious in my view." }-

The entire concept is completely correct. But to take it correct you need to accept it as just another layer of protection. Then this is OK. The concept is no mean can help with a zero-day malware except behaviour based detection system. Idea "once malware is started you are defeated" is wrong. To make something malware needs to do something unusual. And here it will be catched by behavoiur based defence.

[subset]

-{ Quote: "The entire concept is completely correct." }-
Did you ever set "Deactivate HIPS Features" to test the OA firewall?
And did you ever run "Uninstall Firewall" before to test OA HIPS?
Who may win...

The entire concept is completely false and misleading.
It's clearly a HIPS test, not a firewall test.
But why don't they label this tests as HIPS tests?
Evidently because of commercial interests, more companies to molest and a lot more money in sight...
"There are no limits of the frequency of the paid tests."
It is really ridiculous.

Cheers

[wat0114]

Some of you complaining that it should be a HIPS test are missing the point. Matousec has made it abundantly clear in his Design of ideal personal firewall (http://www.matousec.com/info/articles/design-ideal-personal-firewall.php) page his concept of the ideal personal firewall. Read through it and you will see that he feels the ideal personal firewall should include HIPS-like features. It is his opinion only. That is all.

Unlike recently, this does not mean I'm for these leak tests and the race to design the leak proof firewall. These firewalls are just turning into bloated monstrosities, giving the users of them more headaches than they're worth. It's okay maybe for a while to use these products, because they can be useful tools, aiding the user in learning about the inner workings of Windows, but eventually they become tiresome. Even Jetico 2 is getting out of control. They want first place, so they block even the user from Windows Services :(

[Einsturzende]

Ok, again... Personal Firewall must be able to control net request of ALL application and that include: Leak test, malware, rootkit, spyware, viruses... ALL application.
How will that be achieved, I don't care.

[alex_s]

-{ Quote: "Did you ever set "Deactivate HIPS Features" to test the OA firewall?
And did you ever run "Uninstall Firewall" before to test OA HIPS?
Who may win...

The entire concept is completely false and misleading.
It's clearly a HIPS test, not a firewall test.
But why don't they label this tests as HIPS tests?
Evidently because of commercial interests, more companies to molest and a lot more money in sight...
"There are no limits of the frequency of the paid tests."
It is really ridiculous.

Cheers" }-
Why should I disable my protection to run the tests ?
As for entire concept, it depends on what you personally do call Firewall. In case Firewall is only a packet filter for you, then you are right, but in case one regards Firewall as a program that controls all the network activity (as in my case), then you are wrong. Try wikipedia. Packet filters were the first firewalls generation. Now we have at least the third generation and the term meaning also shifted. I mean modern firewall just cannot run without HIPS to comply with the modern requierements. This is mainstream, there is nothing anybody can do with this.

[alex_s]

-{ Quote: "Some of you complaining that it should be a HIPS test are missing the point. Matousec has made it abundantly clear in his Design of ideal personal firewall (http://www.matousec.com/info/articles/design-ideal-personal-firewall.php) page his concept of the ideal personal firewall. Read through it and you will see that he feels the ideal personal firewall should include HIPS-like features. It is his opinion only. That is all.

Unlike recently, this does not mean I'm for these leak tests and the race to design the leak proof firewall. These firewalls are just turning into bloated monstrosities, giving the users of them more headaches than they're worth. It's okay maybe for a while to use these products, because they can be useful tools, aiding the user in learning about the inner workings of Windows, but eventually they become tiresome. Even Jetico 2 is getting out of control. They want first place, so they block even the user from Windows Services :(" }-

Security and convinience are two contradicting things, unfortunately :)

[Escalader]

-{ Quote: "Why should I disable my protection to run the tests ?
As for entire concept, it depends on what you personally do call Firewall. In case Firewall is only a packet filter for you, then you are right, but in case one regards Firewall as a program that controls all the network activity (as in my case), then you are wrong. Try wikipedia. Packet filters were the first firewalls generation. Now we have at least the third generation and the term meaning also shifted. I mean modern firewall just cannot run without HIPS to comply with the modern requierements. This is mainstream, there is nothing anybody can do with this." }-

To all posters:

Well, FWIW I agree in principle with alex_s.

It is clear that different users have different definitions of "FW". That is fine, but wastes gobs of time while posters talk past each other. Maybe it is not a waste.

If user looks at any modern FW SW, say ZA Pro or say CFW or say OA 2, OA 2 + with AV etc they will see many security features that weren't around in the early days. This doesn't mean the past was bad or that the present FW Suites are good or anything like that.

What it does mean is the user needs to KNOW what they need/want security wise on their PC's. That is hand they can look at the latest offerings including the Matousec tests draw their own conclusions and select their tools!

In my case, FWIW I use a set of tools for security. OA 2 has the 2 way FW I want/need and a HIPS integrated with it. It has a bunch of other features as well like web and mail shields that since they come with the suite I use.

I decided not to get the AV from the same vendor since I didn't want all my security eggs in one basket.

In doing a test I would only remove the H/W shields otherwise all you test is them not the SW FW/suite.

The Matouse and all these testing services are good IMO since they make the vendors review their logic and gives their flaws a public airing!

Just my usual rant ;D

[The_1337]

it's stupid if he tests based on "his ideal firewall." how is it fair if companies don't think exactly as he does? the tests just become his opinion on a what a firewall should be.

[alex_s]

-{ Quote: "it's stupid if he tests based on "his ideal firewall." how is it fair if companies don't think exactly as he does? the tests just become his opinion on a what a firewall should be." }-
Fair ? LOL

Any company, any customer and any private person may think whatever he wishes and test whatever he wants with whatever criterias he chooses. This is the only fair idea I know. And everybody is free to develop his own testing program and introduce it to the public (if he is skilled enough and brave enough, of course).

What does prevent you from doing it right ?? Just do it, and everybody will applaud :)

[The_1337]

there is no fair way to do testing, and that's my point.

[alex_s]

-{ Quote: "there is no fair way to do testing, and that's my point." }-

I think it is hardly anybody would argue this statement. Does it mean anybody who tries should be bashed ? :)

My point is a clever person can find useful information is anything, including Matousec tests. I'd say there is a lot of useful information a person can get there. But to do it information should be treated properly. If treated improperly it goes completely meaningless and even misleading. But this is not Matousec fault, this is a person's fault. Matousec is сonsistent. He publishes his methodology, testing conditions, tests themselves and his approach. You can disagree with his approach, but you can hardly find any inconsistency in case you read everything carefully. And I should add, the work he does is VERY timeconsuming and VERY difficult.

[Nebulus]

-{ Quote: "I mean modern firewall just cannot run without HIPS to comply with the modern requierements." }-
There is no such thing as "modern requierements". Security requirements varies from person to person, this is why a general testing methodology like Matousec's is meaningless. Of course, for Matousec everything it's fine because the testing is in accord with his view on firewalls.

-{ Quote: "
This is mainstream, there is nothing anybody can do with this" }-
Really? Well, I can do something about it. I run Sygate (or Kerio 2), both being without HIPS.

[alex_s]

-{ Quote: "There is no such thing as "modern requierements". Security requirements varies from person to person, this is why a general testing methodology like Matousec's is meaningless." }-

There are. Modern firewall has to treat modern network-related malware in a safe way. This is the main modern requirement. Also modern firewall should be easy to use by a non-technically skilled average user (computers are now not only IT tools as it was not too long ago).

[Einsturzende]

-{ Quote: "There is no such thing as "modern requierements". Security requirements varies from person to person, this is why a general testing methodology like Matousec's is meaningless. Of course, for Matousec everything it's fine because the testing is in accord with his view on firewalls.


Really? Well, I can do something about it. I run Sygate (or Kerio 2), both being without HIPS." }-
Personal Firewall must distinguish which application made net request, that is now like it was before, if it cant it is useless. ← period

[Mrkvonic]

Hello,
Why does a firewall have to distinguish what application made the net request?
Mrk

[alex_s]

-{ Quote: "Hello,
Why does a firewall have to distinguish what application made the net request?
Mrk" }-

http://en.wikipedia.org/wiki/Firewall

===
Firewall technology emerged in the late 1980s when the Internet was a fairly new technology in terms of its global use and connectivity. The predecessors to firewalls for network security were the routers used in the late 1980s to separate networks from one another [1]. The view of the Internet as a relatively small community of compatible users who valued openness for sharing and collaboration was ended by a number of major internet security breaches, which occurred in the late 1980s.[1]:
Clifford Stoll's discovery of German spies tampering with his system [1]
Bill Cheswick's Evening with Berferd" 1992 in which he set up a simple electronic jail to observe an attacker[1]
In 1988 an employee at the NASA Ames Research Center in California sent a memo by email to his colleagues[citation needed] that read,“ We are currently under attack from an Internet VIRUS! It has hit Berkeley, UC San Diego, Lawrence Livermore, Stanford, and NASA Ames. ”

The Morris Worm spread itself through multiple vulnerabilities in the machines of the time. Although it was not malicious in intent, the Morris Worm was the first large scale attack on Internet security; the online community was neither expecting an attack nor prepared to deal with one.[2]

[edit]
First generation - packet filters

The first paper published on firewall technology was in 1988, when engineers from Digital Equipment Corporation (DEC) developed filter systems known as packet filter firewalls. This fairly basic system was the first generation of what would become a highly evolved and technical internet security feature. At AT&T Bell Labs, Bill Cheswick and Steve Bellovin were continuing their research in packet filtering and developed a working model for their own company based upon their original first generation architecture.

Packet filters act by inspecting the "packets" which represent the basic unit of data transfer between computers on the Internet. If a packet matches the packet filter's set of rules, the packet filter will drop (silently discard) the packet, or reject it (discard it, and send "error responses" to the source).

This type of packet filtering pays no attention to whether a packet is part of an existing stream of traffic (it stores no information on connection "state"). Instead, it filters each packet based only on information contained in the packet itself (most commonly using a combination of the packet's source and destination address, its protocol, and, for TCP and UDP traffic, which comprises most internet communication, the port number).

Because TCP and UDP traffic by convention uses well known ports for particular types of traffic, a "stateless" packet filter can distinguish between, and thus control, those types of traffic (such as web browsing, remote printing, email transmission, file transfer), unless the machines on each side of the packet filter are both using the same non-standard ports.

[edit]
Second generation - "stateful" filters
Main article: stateful firewall

From 1980-1990 three colleagues from AT&T Bell Laboratories, Dave Presetto, Janardan Sharma, and Kshitij Nigam developed the second generation of firewalls, calling them circuit level firewalls.

Second Generation firewalls in addition regard placement of each individual packet within the packet series. This technology is generally referred to as a stateful firewall as it maintains records of all connections passing through the firewall and is able to determine whether a packet is either the start of a new connection, a part of an existing connection, or is an invalid packet. Though there is still a set of static rules in such a firewall, the state of a connection can in itself be one of the criteria which trigger specific rules.

This type of firewall can help prevent attacks which exploit existing connections, or certain Denial-of-service attacks.

[edit]
Third generation - application layer
Main article: application layer firewall

Publications by Gene Spafford of Purdue University, Bill Cheswick at AT&T Laboratories, and Marcus Ranum described a third generation firewall known as an application layer firewall, also known as a proxy-based firewall. Marcus Ranum's work on the technology spearheaded the creation of the first commercial product. The product was released by DEC who named it the DEC SEAL product. DEC’s first major sale was on June 13, 1991 to a chemical company based on the East Coast of the USA.

The key benefit of application layer filtering is that it can "understand" certain applications and protocols (such as File Transfer Protocol, DNS, or web browsing), and it can detect whether an unwanted protocol is being sneaked through on a non-standard port or whether a protocol is being abused in a known harmful way.

[edit]
Subsequent developments

In 1992, Bob Braden and Annette DeSchon at the University of Southern California (USC) were refining the concept of a firewall. The product known as "Visas" was the first system to have a visual integration interface with colours and icons, which could be easily implemented to and accessed on a computer operating system such as Microsoft's Windows or Apple's MacOS. In 1994 an Israeli company called Check Point Software Technologies built this into readily available software known as FireWall-1.

The existing deep packet inspection functionality of modern firewalls can be shared by Intrusion-prevention systems (IPS).

Currently, the Middlebox Communication Working Group of the Internet Engineering Task Force (IETF) is working on standardizing protocols for managing firewalls and other middleboxes.

[Bunkhouse Buck]

-{ Quote: "The entire concept is completely correct. But to take it correct you need to accept it as just another layer of protection. Then this is OK. The concept is no mean can help with a zero-day malware except behaviour based detection system. Idea "once malware is started you are defeated" is wrong. To make something malware needs to do something unusual. And here it will be catched by behavoiur based defence." }-

That is why I use software that detects behavior changes. You did not refute what I argued about firewalls- if anything, you reinforced it with your comments.

[Mrkvonic]

Hello,
alex, I can read wikipedia myself. I still do not have your answer. Why should a firewall need to tell what process made the request?

Now, my explanation:

Application-based filtering is good for a TRUSTED environment only. For instance, I use firewalls to restrict Windows services from outbound connections, knowing these are legitimate applications. They will not try to do anything more than what the firewall reports. What you see is what you get.

If you have "malware" on your machine and this malware has access to the kernel, it can effectively reroute process IDs or process calls, making any driver or application running on top of it (including firewall) think it's executing a legitimate application.

A subverted kernel cannot be trusted, hence all and any internal identification becomes meaningless. This is why leaktests have no meaning.

You might install some simple malware that will not try to outsmart your firewall and then the firewall will be effective. But then, you might install something that rewrites half the kernel and from that moment on, nothing is as it ever seems to be. Not only does your firewall become useless - and remains silent - everything else is changed, too.

Leaving the chances of existence of either aside, as well as the ways how one gets infected or infects himself or herself, it is impossible to protect the operating system from itself.

Any system where any application has FULL control - like in Windows - the kernel can be changed and manipulated.

The firewall might work 50%, 90%, 99.3% of times, but there are times when it might not. Testing leaktests in a controlled environment, where you know when and what to expect does not reflect the reality where the user downloads something and then starts to install and begins clicking yes yes yes.

Mrk

[Einsturzende]

-{ Quote: "Hello,
Why does a firewall have to distinguish what application made the net request?
Mrk" }-
I said Personal firewall
because it is personal fw job.
Your question is dumb, like this one: Why media player playing video?

[Bunkhouse Buck]

-{ Quote: "Hello,
alex, I can read wikipedia myself. I still do not have your answer. Why should a firewall need to tell what process made the request?

Now, my explanation:

Application-based filtering is good for a TRUSTED environment only. For instance, I use firewalls to restrict Windows services from outbound connections, knowing these are legitimate applications. They will not try to do anything more than what the firewall reports. What you see is what you get.

If you have "malware" on your machine and this malware has access to the kernel, it can effectively reroute process IDs or process calls, making any driver or application running on top of it (including firewall) think it's executing a legitimate application.

A subverted kernel cannot be trusted, hence all and any internal identification becomes meaningless. This is why leaktests have no meaning.

You might install some simple malware that will not try to outsmart your firewall and then the firewall will be effective. But then, you might install something that rewrites half the kernel and from that moment on, nothing is as it ever seems to be. Not only does your firewall become useless - and remains silent - everything else is changed, too.

Leaving the chances of existence of either aside, as well as the ways how one gets infected or infects himself or herself, it is impossible to protect the operating system from itself.

Any system where any application has FULL control - like in Windows - the kernel can be changed and manipulated.

The firewall might work 50%, 90%, 99.3% of times, but there are times when it might not. Testing leaktests in a controlled environment, where you know when and what to expect does not reflect the reality where the user downloads something and then starts to install and begins clicking yes yes yes.

Mrk" }-

Finally someone/something I agree with in over a year in this forum. :thumb: There is more paranoia and nonsense about security in this forum than any other place I have seen. So-called outbound protection from a firewall is a myth if your operating system has been compromised. The kernel can be changed and manipulated.

[Mrkvonic]

-{ Quote: "I said Personal firewall
because it is personal fw job.
Your question is dumb, like this one: Why media player playing video?" }-

Hello,

No it's not.

Media player - plays media
Firewall controls - ?

Firewall is also an application so ... but it can also control packets ...

Now, questions cannot be dumb. Only answers can.

Mrk

[Einsturzende]

Personal fw is not just packet filter, it can also distinguish which app. made call,
Like media player can play music and movies...::)

[Mrkvonic]

Hello,
It MAY distinguish - provided the kernel has not been compromised ...
That's where the ENTIRE problem rests.
Mrk

[wat0114]

-{ Quote: "
The firewall might work 50%, 90%, 99.3% of times, but there are times when it might not. Testing leaktests in a controlled environment, where you know when and what to expect does not reflect the reality where the user downloads something and then starts to install and begins clicking yes yes yes.

Mrk" }-

With all due respect, the same can be said about anitiviruses; they work usually < 100% of the time. I side somewhat with you and Bunkhouse, but I also side in part with alex. In the right hands, a personal firewall can be an invaluable security tool, provided, of course, that the malware does not completely circumvent it. You mention the user will click yes, yes, yes, but a knowledgeable, responsible user will probably see that during installation of unknown malware, application whoopie.exe is attempting to connect to remote port 6666, therefore having the opportunity to stop it from connecting out. Maybe the system is toast because the malware is installing, but at least it does not transmit personal data.

As I've mentioned earlier, I do believe these firewalls, some of them at least, are getting out of control. I don't mind basic, solid application control incorporated in the firewall, but the race to keep up with every conceivable leaktest is causing some of these products to bulge at the seams, so to speak ;)

[Einsturzende]

-{ Quote: "Hello,
It MAY distinguish - provided the kernel has not been compromised ...
That's where the ENTIRE problem rests.
Mrk" }-
Oh, if kernel is compromised then everything is compromised...
Personal FWs should have rootkit installation detection and other similar HIPS like tools for not to be kernel compromised and thus to be able to control app net requests properly, so I hope Matousec will implement more of rootkit like and similar test in the future...

[Pedro]

There are several things to look at, and at the end i'm with wat0114.

You can take the problem apart by:
-what users can actually interpret and use these behavior alerts;
-how well does the tool report it, is it understandable;
-how well can the tool detect the various dangerous actions - Matousec's tests;
-if the user is on a LUA, the kernel is likely intact;

on a different note:
-tell me a story on why/how would i execute malware;
and oh so on.

The tests are interesting for who ever finds them interesting.
Interpreting the results like they prove xyz firewall is poor is the wrong way of looking at it. If the program doesn't aim to detect malware behavior, then the tests will only show that.

Of course, one can say that it's exactly that kind of conclusion the website suggests.
I don't think i can argue with that. It would be best to place a note everywhere on the website, or point out which products aren't built for all that. They have a more specific use.

On a side note, it seems no one noted the network performance impact tests.
I believe it was the main reason for the OP's post, but i could be wrong..

[alex_s]

-{ Quote: "If you have "malware" on your machine and this malware has access to the kernel" }-

My firewall will not allow it to access the kernel. It will do it to be able to fulfil its main task -- to control network traffic. Cicle is closed. Concept is still correct.

[alex_s]

-{ Quote: "on a different note:
-tell me a story on why/how would i execute malware;
and oh so on." }-

A lot of reasons. Childrens, from email received by your Mom, from a flash drive with students' tasks you need to check. This is what did come to my mind immediately.

[Pedro]

Well, are you saying the child will answer the prompts?
If you're thinking about password protection, why do you need anything other than execution control?

Take SSM free (or Kerio's HIPS, whatever). I use it just to have visible control on processes. I know, or i think i know, nothing will execute if i didn't say so.
I disconnect UI. Why does SSM need to monitor all those things, if it's about controlling children, e-mail attachments, etc?

I admit there's plenty i don't know, but so far no one has illustrated why simple execution control won't cut it, like other members say.

[alex_s]

-{ Quote: "I don't mind basic, solid application control incorporated in the firewall, but the race to keep up with every conceivable leaktest is causing some of these products to bulge at the seams, so to speak ;)" }-

So, do you think there is no need in additonal security layers or do you want to say that IPS part must be separated from firewall ? In the latter case all the logic cries that separated applications will take more resources than a single one with a shared data and code for the different tasks.

[alex_s]

-{ Quote: "So-called outbound protection from a firewall is a myth if your operating system has been compromised. The kernel can be changed and manipulated." }-
One of the main tasks of the HIPS part is to prevent the kernel compromise. BTW, you miss some important point in the tests. The main part of many tests is to prove that unknown program (presumably malware) can compromise some system process (by dll inject or memory tampering, for example) and once this happened the way to the kernel is open. For example it is easy to catch unknown program attempt to lauch a kernel driver, but this is much harder in case the same action is executed by svchost or winlogon or system. Or to be fair it can be catched in any case, but alerting on every action of every program makes a system completely unusable. So I'd added to the score another parameter - the number and meaningfulness of the alerts. Though, this part is very difficult to formalize. Another lack of the tests is the absence of the rootkit teckniques (kernel intrusion). I think this will be added at some time. But even with all these lacks the tests are not completely meaningless, they are just not comprehensive. In any case there are not any comprehensive tests available currently. But extra information never harmed to thoughtful person to get more accurate picture than just based on pure speculations and very limited personal experience.

[Mrkvonic]

Hello,

And all your theories go to water when you put happy-click-joe in front of the computer and then he has to answer the prompts. OLE, DLLs, whatnots, happy happy click.

HIPS / Leaktests may be interesting to 0.019% of population, even less. And they are about as practical. You might as well use Linux and be done with it, if you want real control.

Mrk

[wat0114]

-{ Quote: "So, do you think there is no need in additonal security layers or do you want to say that IPS part must be separated from firewall ? In the latter case all the logic cries that separated applications will take more resources than a single one with a shared data and code for the different tasks." }-

The need for more security layers depends on the individual(s) using the machine. A knowledgeable, careful and responsible person really does not need too much in terms of security. I can say this simply based on my own experience. As for a separate application handling the IPS part of things, this would depend on the user's needs as well as the personal firewall being used. I have found the last 4.0 version of Outpost and the recent 2.0.x versions of Jetico 2 to be very good on their own, without the need to add a separate IPS layer. However, I have found the very recent version of Jetico 2 and the newer versions of Outpost to be too buggy to my liking, probably because they are trying to address too many leak POC exploits.

[Dwarden]

so in the end the safest thing is use your self coded kernel :)
and just hope noone develops kerneless malware able to infest any hw :)

[alex_s]

-{ Quote: "Hello,

And all your theories go to water when you put happy-click-joe in front of the computer and then he has to answer the prompts. OLE, DLLs, whatnots, happy happy click.

HIPS / Leaktests may be interesting to 0.019% of population, even less. And they are about as practical. You might as well use Linux and be done with it, if you want real control.

Mrk" }-
Just instruct him to press "block" in case he doesn't inderstand the word :)

[alex_s]

-{ Quote: "The need for more security layers depends on the individual(s) using the machine. A knowledgeable, careful and responsible person really does not need too much in terms of security. I can say this simply based on my own experience. As for a separate application handling the IPS part of things, this would depend on the user's needs as well as the personal firewall being used. I have found the last 4.0 version of Outpost and the recent 2.0.x versions of Jetico 2 to be very good on their own, without the need to add a separate IPS layer. However, I have found the very recent version of Jetico 2 and the newer versions of Outpost to be too buggy to my liking, probably because they are trying to address too many leak POC exploits." }-
I think they have no choice. I can hardly imagine security vendor advocating security hole (which leaktest POCs demonstrate). Vendor should admit that either he is unable to treat the hole in secure way or he is unable to develop security software in a professional way.

[Mrkvonic]

-{ Quote: "Just instruct him to press "block" in case he doesn't inderstand the word :)" }-

Exactly!!!!
You don't need HIPS for that. That's exactly my point!
What doesn't get executed can't hurt you.
Mrk

[alex_s]

-{ Quote: "Exactly!!!!
You don't need HIPS for that. That's exactly my point!
What doesn't get executed can't hurt you.
Mrk" }-
I'm not ready to restrict myself from running everything unknown just not to get compromised ! I rely on my HIPS and my experience and feel myself safe to surf and download and start a lot of things. This is my way. I understand that other people may think other way. I do not argue _their_ way.

[wat0114]

-{ Quote: "I think they have no choice. I can hardly imagine security vendor advocating security hole (which leaktest POCs demonstrate). Vendor should admit that either he is unable to treat the hole in secure way or he is unable to develop security software in a professional way." }-

Vendors will probably never admit that, the latter part especially :)

Is there anyone who can honestly say there is a product out there that attempts to adhere to Matousec's requirements that isn't buggy in some way? There are some products out there such as Outpost, Comodo, Jetico and Online Armor that do a noble job of mirroring Matousec's requirements, but there seems to be no shortage of never-ending posts on these products from people who are encountering "bug-like" issues with them.

They're probably doing their best to develop a perfect, leakproof and stable product, but it's a very complex endeavor. There seems to always be some leaktest flavor of the month that gets past these products, then the patching continues once again. It's endless.

[alex_s]

-{ Quote: "Vendors will probably never admit that, the latter part especially :)

Is there anyone who can honestly say there is a product out there that attempts to adhere to Matousec's requirements that isn't buggy in some way? There are some products out there such as Outpost, Comodo, Jetico and Online Armor that do a noble job of mirroring Matousec's requirements, but there seems to be no shortage of never-ending posts on these products from people who are encountering "bug-like" issues with them.

They're probably doing their best to develop a perfect, leakproof and stable product, but it's a very complex endeavor. There seems to always be some leaktest flavor of the month that gets past these products, then the patching continues once again. It's endless." }-

Yes, life is life. Everything moves and changes all the time. New code -- new bugs, etc etc :)

I would just added, that they not only try to pleasure Matousec. In the past I have sent every rootkit, keylogger, unhooker and malware I could get to bypass OA to Mike and all of them were addressed. Comodo issued new leaktests and they also were addressed. Apt kill7 was addressed. NiCM tests were addressed. Matousec just seems to be the most consistent and popular tester, this is why he "seems" to be most addressed (and bushed :) )

[hammerman]

-{ Quote: "Vendors will probably never admit that, the latter part especially :)

Is there anyone who can honestly say there is a product out there that attempts to adhere to Matousec's requirements that isn't buggy in some way? There are some products out there such as Outpost, Comodo, Jetico and Online Armor that do a noble job of mirroring Matousec's requirements, but there seems to be no shortage of never-ending posts on these products from people who are encountering &quot;bug-like&quot; issues with them.

They're probably doing their best to develop a perfect, leakproof and stable product, but it's a very complex endeavor. There seems to always be some leaktest flavor of the month that gets past these products, then the patching continues once again. It's endless." }- I agree with this 100%. I have tried most of the top firewalls and have been really disappointed with performance and stability. I think too much time has been spent trying to pass Matousec's tests and packing in as much functionality as possible and not enough on producing rock-solid , high performance, user-friendly software. I really wish that firewall vendors would concentrate on these issues rather than passing all the latest tests. Perhaps it would help if Alex would stop sending OA all his rootkits, keyloggers and malware. I would add a smile at this point but I think my firewall is playing up today!!

[alex_s]

-{ Quote: "I agree with this 100%. I have tried most of the top firewalls and have been really disappointed with performance and stability." }-
There is a lot of the old stable firewalls out there. Just take one of them.
-{ Quote: "Perhaps it would help if Alex would stop sending OA all his rootkits, keyloggers and malware." }-
No and never.

I'm ready to sacrifice some stability to the power. This is why I don't use old stable firewalls and this is why I use windows (and used to do it even when it was VERY VERY unstable and buggy) :)

[hammerman]

-{ Quote: "There is a lot of the old stable firewalls out there. Just take one of them.

No and never.

I'm ready to sacrifice some stability to the power. This is why I don't use old stable firewalls and this is why I use windows (and used to do it even when it was VERY VERY unstable and buggy) :)" }-

Hi Alex,

The comment about not sending malware etc to OA was definitely tongue-in-cheek and thought it might be recognised as such. Carry on the good work.

Are you saying that if I want a stable firewall, I need to choose an old one? I am sure the modern firewall vendors aspire to producing high stability, high performance firewalls. They just haven't quite got there yet and the need for passing the latest leak test is a distraction.

[alex_s]

-{ Quote: "Hi Alex,

The comment about not sending malware etc to OA was definitely tongue-in-cheek and thought it might be recognised as such." }-
Shame on me ! I have no justification :)