AntiLeak racing insanity?

Matousec Tests, have definitily brought insanity in the firewall competition.

Do not get me wrong, I continue to believe that some basic antileak capability is necessary mainly for giving some basic information/instracting the user on how the windows applications interract with its other. This can teach the users to avoid installing cracks or at least, if they continue to use them to try isolating them as possible. Also it can help the people to be more aware that even legitimate products phone home, without their explicit permition (most of them do not include this info, not in the help and not in the EULA).

But all this thing it has gotten out of hand. Instead of firewalls now we have a bunch of hips applications with basic firewall ability. Hips are good for the people that can understand their various warnings and know how to troubleshoot their computer in case theymake a wrong choice. They are not designed for the majority of the users.

Firewall are transformed in powerfull hips applications that leave the users unprotected; not because they do not work as they should but because most of the users do not know what to answer at the various pop-ups. So the users give wrong answers and they toggle the protection or they acccidentally lockup their programs and in the worst case their pcs.

For not talking about the various incompatibilities, the BSODs that they are causing and the constant access on the hard disks.

1-2 years ago, I was helping the others to clean their pcs from malware. Now I help the others to "clean" the mess that themselves have created with the "help" of the antimalware/hips applications (antiviruses, antispywares, firewalls).

Matousec's firewall challenged is only a hips challenged. Nothing more and nothing else. It has nothing to do with their firewalls capabilities.

The users are the only ones that do not benifit from those tests. The only ones that benifit are matousec and the security vendors. It is only a great pubblicity system that has nothing to do with the filtering of the network traffic.

The best firewall according to the matousec tests, would be faronics AE (or any other antiexecutable application).

ps. just a small rant

Panagiotis

 

I pretty much feel the same way.


Regards,
Phant0m``

 

Hi

I agree 100% too.

I think what the firewall vendors really need to do is get together and all agree that Matousec is getting too extreme now and stop participating in the tests.

 

Good observation

 

You have said it very well. Such results begin to mislead users with those POC results that gives negative images to firewalls rated in their "protection level". In my opinion, those POC leaktests shouldn't be given too much significance in a way that their results would also be the same when tested with real malware. Shouldn't it be somewhat doubtful for this leaktests to perfectly replicate complex malware behavior? Malware writers should have begun doing a major overhaul of methods in trying to bypass outbound protection of firewalls if those POC leaktests had just perfectly duplicated malware behavior. A true malware may not be completely compared with POCs. There would always be a difference, either major or minor, between the two.

 

Great post!

I suspect that many vendors are simply hard coding against the specific "leak test" and not the actual threat anyway so they will pass.

 

So everyone agrees. How boring! We need someone who is absolutely convinced that having HIPS features in a firewall is a must and that any firewall that isnt completely leakproof is useless and the developers need to be named and shamed for their poor programing!

 

firewall leak testing is nothing but HIPS testing
all the advanced firewalls like comoda , OA ect.. are nothing without their hips module

evidences of my opinion

1- currently the 2nd place in matousec firewall challenge is "prosecurity" 93% which is not even a firewall , and i think this is funny
also safety system monitor "hips only " is ranked as "good
while most of the pure firewalls marked as "none"

2-in comodo 3 with the D++ disabled , it's very very easily killable by the task manager , this the point which inforced comodo vendor to maintain some of the D+ activated even if u choosed the basic firewall without hips
can be very easily terminated by the weakest trojan in the world
so comodo without its hips module , is weaker than the built in windows xp firewall

3- such leak test challenge measure the hips power of the firewall , and ignore completely other essential aspects such as

A) flexability , usability and ease of use
B) bugs and conflicts with othe softwares "some firewalls like outpost and comodo sometimes their bugs cause problems "eg. BSODs" which may be worse than a virus attack or those that can be caused by a hacker on ur pc
c) memory usage and slow boot time" like zonalarm and OA "


4-if u are using a AV
so the alarm will be , virus detected
and this is very easy , does not need advanced user
and even ur kid will simply answer " delete"

but the hips protection in such firewalls are fully user dependent , they only need a very advanced and experienced user
u can face alarm xxx.exe is trying to do xxx process
is this normal or not
i don't know
should i allow
should i deny
can i call my experienced friend at home and ask him this difficult Question
so , what will u do if this alarm appeared hundreds of times every day
will u call him 100 times


any thing which is user dependent is liable to mistakes
it's very easy that u may answer a Q , wrong answer

tha lazy click syndrome ,
u just do nothing but
allow
allow
allow
allow
allow

may be between these "allow"s , u did a mistake
may be u allowed a trojan to connect to outside ur pc
the firewall gave u the alarm , but u "or even ur kid" answered the wrong answer and u got hacked "theoritically" in the presence of the number one firewall in the leak test ranking


and sometimes i heard a funny comment from a comodo user who get bored form ansering such hundreds of alarms every now and then
he said " if i really want to be bothered , i'll simply go downstairs and talk to my wife" better than keep answering such endless Question alarms


the fact simply is "suck leak tests measure the ability of the hips module in the firewall to alarm u about certain process in ur pc , and in the same time they ignore completely , the quality of the user's answer , is there's a possibility that the user answer the Q by the wrong answer ? , sure it's possible

 

Not to mention that CFP 3.0 which is supposed to be the best in Matousec's testing failed against programs with stolen rights to connect to the internet and also failed to block 2 Trojans of 10 of them from phoning home (by "PC Welt" magazine)-again real stuff some leak-tests.

 

There is a "Rogue Software" list. Someone needs to start a list of "Rogue Security Software Tests", with Matousec's nonsense first on the list.

 

I have been saying leak testing is a waste of time for years. Most users would be fine with the windows built in firewall, especially the one in Vista.

If you want to see how badly some of these HIPS enabled firewalls muck up your system, run GMER with and without the firewall installed. Even with D+ turned off, Comodo 3 hooks into every running program multiple times.

You might also check your commit charge with and without the firewall installed (not applicable to Vista) and then run a CPU benchmark like SuperPi on a with and without basis.

 

This is all about marketing. Same spiel that Bill Gates has been pulling since the 1980's.

 

pandlouk deserves a cookie.

 

As the creator of System Shutdown Simulator (SSS),
I believe leaktests are important in testing firewall and HIPS.
In the future we may add Sandbox leaktests...

Are people in this thread saying they don't want further leaktests... You have no interest whatsoever in SSS?

So I'll just disappear...

 

don't disappear .. I think if people really think about it they would like the leak tests .. I believe it is more a reaction to matousec. Enougth people will refer to there results that vendors can not ignore them. But I belive they put way to much importance to the leak test and not enougth to the underlying firewall.

Adding things like keyloggers, which have nothing to do with firewalls in my opion. I would expect to score a firewall it should be something like

15% kill ability
15% leak ability
20% performance
50% Network/application control ..

where with the matousec results it is like 60% leak, 35% kill ability ...

 

Agree. Matousec is a waste of time. Ignore <click>

 

dmenace, antileak tessting in essential, but not, when it misleads people people by thinking, that it applies to firewall only. If he would rename it to antimalware aplications testing, then it would be adequate, so far it proofs only one thing, that firewall are useless as long as you have a good HIPS, which prevents malware even to start, so it can not connect to the internet. I myself am happy with a poor Windows Firewall and with no malware, nor antimallware aplications in my PC.

 

I understand your views precisely especially regarding leaktests being more for HIPS software than firewalls.

This is particularly relevent when matousec classifys SSM as a firewall when it has no inbound filtering at all.

Its just that when a user is behind a SPI/NAT firewall inbound performance is a bit useless... hence the focus on outbound performance.

However I totally agree that I would rather have a firewall with advanced network control / logging capabilities rather than leaktest performance.

An interesting debate!

 

I can imagine the whole thing with leaktests started when router boxes took charge of the inbound protection. These come now with NAT, SPI and whatnot, so firewall vendors had to concentrate on the aspect of security a router could not touch - application outbound control (well, they have to make their living too). As this involved executable injections, many came with various PoCs to challenge this approach. Matousec collected these PoCs and conducted some tests, that is all. It is up to you to interpret it right.

No. It's the other way round - it is not Matousec's fault that the users are generally ignorant and cannot distinguish between HIPS and firewall features. As it goes for any other test, a certain level of knowledge is needed to interpret it right. A test is a test, not an education tool. If you don't understand it either learn or steer clear.

Cheers,

 

Solution: Besides Leak testing, make a new testing consisting in:
Putting a firewall in a system, gathering some hackers, trying to get access into the system using different ways, seeing how the firewall reacts

 

I agree, leaktests are important for both firewalls and hips.

But for the firewall part there must be involved also somekind of network traffic.

Matousec is misleading the users. For example SSM, comodo (with the firewall component disabled), etc. will pass those tests with flying colors.
So where does it stand the title "firewall challenge"? It is totally unfair to the other firewalls (which might do a far better job in filtering the network traffic).

 

Hello,

This has been done many times - it's boring.

Remember the Vista, Mac, Ubuntu test? Well, no OS fell for brute hacking.
The TCP/IP protocol has pretty much been unchanged since 1980, so it's rather boring ...

Mrk

 

But if you consider that wifi connections are becoming the standard, personal firewalls become more necessary than ever.

My thoughts are exactly the same. Network control and extend logging should be the primary fuction of a firewall.

 

I fully agree with this.
I emphasize 0% keyloggers.