New tests from Matousec

In case someone missed these ...

2008-05-06: Three new tests have been added to the suite. PerfTCP and PerfUDP have been added to Level 1, SockSnif to Level 8.
2008-04-24: Seven new tests, namely Keylog1, Keylog2, Keylog3, Keylog4, Keylog5, Keylog6 and Keylog7, have been added.

http://www.matousec.com/

i would like to avoid the 'endless' discussion about Matousec group motives or style so please comment the tests and methology self ...

 

Hi,

a comment on the tests.
Do vendors have to pay another time for the added tests to be tested on already tested programs?
Or will this new tests be tested and added for free to the tests of already tested programs?
Just for curiosity.

Cheers

 

I don't understand why he tests pure firewall programs as if they were hips. Or why he tests only the firewall from security suites. Of course they aren't going to do well. This guy needs to call it the hips challenge and stop testing stand alone firewalls.

 

I could not agree more.

 

Hi,

this may sound dumb, but why is for example the AV modul from Avira Suite forbidden and has to be shut down during the tests and the HIPS module from Online Armor is allowed to run during the tests.

As said before before, this tests are determined as "Firewall Challenge" by the vendor.
But on one side blacklisting the tests for AVs is forbidden and on the other side HIPS are not forced to add the tests to their whitelist.
Or at least AV modules and HIPS modules have to be shut down during the tests for a real "Firewall Challenge".

Cheers

 

Default settings?

 

Firewalls and HIPS will react to both known and unknow malware, unlike AVs which only reacts again known (or detected by heuristic) malware. Also testing with the AV enabled would be meaningless as every vendor would add the tests to the blacklist and get 100% in 5 seconds, without increasing protection against real malware using those techniques.

Nowadays most firewalls are not pure firewalls but have some HIPS components. If you disable the HIPS in a firewall you should disable almost every feature in the rest, because they are also HIPS. Almost every leaktest would fail if you disable these features.

 

that's described in detail on Matousec website and was discussed to death on this and comodo forums x times etc...

 

Does this mean these tests are meaningful because AVs are forbidden
Either they test programs like they are or they test just firewalls only.

Now this are just crude tests, far from reality because of their strange settings and without any value for users.

Reality would perhaps look like this (again for example Avira and OA):
- Avira's heuristic might detect most of the files as HEUR/Malware, TR/Hijacker.Gen or TR/Proxy.Gen etc. >>> user is mostly well-protected.
- Online Armor HIPS might alarm the user with multiple popups about bad things going on. >>> user is mostly well-protected.

So my conclusion is, these tests are pointless because the methology is apparently made between the Ivory Tower and the Temple of Mammon.

Cheers

 

1) Unlike a firewall or HIPS, if the AV (through signatures or heuristics) detects the leaktest it doesn't mean that will detect all malware that uses the same method
2) Testing AVs is quite different than testing HIPS or firewalls. Also there are other well-known AV testing organizations.
3) HIPS are much closer to firewalls than AVs to firewalls. An Antivirus analyses the code of the programs and is based mainly on blacklisting. Firewalls and HIPS monitor the behaviour of the program and are based on whitelisting.

 

hips isnt a firewall either so i dont see why that would be included. i mean it's called a firewall challenge not firewall and hips challenge.

 

But personal firewall isn't complete if cannot distinguish which process made net request, so some technique similar to HIPS must be implemented (and tested of course)...

 

Matousec is a waste of time.. ignore them. The only products that get a 100% percent on those tests are products that are not usable in the real world.

 

write better tests and share source ... same like Matousec does

 

Uhhhhhh... yes they are usable in the real world

 

Comodo and Online Armor are not usable in the real world?
Thanks for that info, until now, I was convinced opposite.

And using both of them (comodo first, oa now) to my complete satisfaction. And based upon my long term experience calling both of them really the best firewalls available !!!

 

Hi,

kindly note that using Comodo FP or Filseclab PF or whatever does not make Matousec's tests and methology meaningful or meaningless per se.

However, the only noticeable result from this "Firewall(?) Challenge" is strangewise:
You can estimate, that an application has some sort of HIPS(!) features or not.

You can not even estimate, that an application has a Firewall included (ProSecurity...).

Blank refusal, try again, fail better.

Cheers

 

I think you'll find that most people, regardless of their opinion on Matousec, are tired of discussing the whole HIPS or FW, or whatever.

Now he comes with some tests that should interest everyone:

So instead of pushing the same arguments back and forth, we could try these?

Cheers

 

I think "Accessing Network" thingy in it is good enough for testing it like an "firewall" and with tests arsenal Matousec uses for testing FWs.

 

Maaaaybee by a small fraction of computer users. I can bet most people even on this newsgroup dont even know what an IP address is, so even the simplest alerts that say "IP address" are totally meaningless to 99.999% of users. Comodo and others ofcourse take the meaningless alerts to a whole new level.

 

On the contrary...

 

SockSniff, sounds nasty.

I have always felt the whole concept of detecting malware by waiting for it to phone home is nonsense.

What is needed are ways to block malware installation in the first place, or at least detect its presence without relying on signatures. Products like Threatfire are a step in the right direction, but are probably not all that reliable yet.

 

99.99999& of users is a over exaggeration. Yes, a vast numbers of users don't know about IP addresses, but the firewalls are still usable by people. Even if they do not know what an IP address is, they can still use it by using an Allow All option rather than just allowing the IP address.

I wouldn't say the software which scores well on leaktests are "not usable".

The thing is, at the end of the day, with the present technology, its still possible for malware to be installed in the first place and leak data or automatically download data, and a leak proof firewall is a method to plug this hole for many users.

 

It is nonsense. Once they are in your house, they can figure out a way to steal your stuff. The entire anti-leak concept is fallacious in my view.

 

Hello,

Like I've said a kabullion times, leaktests only show how easily or in how many different ways the operating system can be fooled. Since the firewall is installed on top of the kernel, it seems like a self-defeat logic to try to control that kernel. After all, the firewall will do only whatever the kernel decides to let it see and process.

Unless the firewall becomes some sort of super service for the kernel, which is absurd. You might as well try a different kernel.

Furthermore, leaktests are ineffective, because:
- People test them deliberately, knowing what to expect. It's easy blocking something called thermite or whatever, but what about something like internet explorer or explorer or svchost.
- They assume you have been infected, which is the worst thing you can possibly do; like drinking poison and then testing if your liver can take it.

Finally, a very good piece of malware will subvert the kernel, change the tcip stack etc - you will get precisely 0 prompts from your firewall.

Mrk