New tests from Matousec

Discussion in 'other firewalls' started by Dwarden, May 7, 2008.

Thread Status:
Not open for further replies.
  1. Dwarden

    Dwarden Registered Member

    Joined:
    Apr 11, 2003
    Posts:
    176
    Location:
    Czech Republic
    In case someone missed these ...

    2008-05-06: Three new tests have been added to the suite. PerfTCP and PerfUDP have been added to Level 1, SockSnif to Level 8.
    2008-04-24: Seven new tests, namely Keylog1, Keylog2, Keylog3, Keylog4, Keylog5, Keylog6 and Keylog7, have been added.

    http://www.matousec.com/

    i would like to avoid the 'endless' discussion about Matousec group motives or style so please comment the tests and methology self ...
     
  2. subset

    subset Registered Member

    Joined:
    Nov 17, 2007
    Posts:
    825
    Location:
    Austria
    Hi,

    a comment on the tests.
    Do vendors have to pay another time for the added tests to be tested on already tested programs?
    Or will this new tests be tested and added for free to the tests of already tested programs?
    Just for curiosity. :cautious:

    Cheers
     
  3. Makav3l1

    Makav3l1 Registered Member

    Joined:
    Nov 26, 2007
    Posts:
    241
    I don't understand why he tests pure firewall programs as if they were hips. Or why he tests only the firewall from security suites. Of course they aren't going to do well. This guy needs to call it the hips challenge and stop testing stand alone firewalls.
     
  4. LoneWolf

    LoneWolf Registered Member

    Joined:
    Jan 2, 2006
    Posts:
    3,408
    I could not agree more.
     
  5. subset

    subset Registered Member

    Joined:
    Nov 17, 2007
    Posts:
    825
    Location:
    Austria
    Hi,

    this may sound dumb, but why is for example the AV modul from Avira Suite forbidden and has to be shut down during the tests and the HIPS module from Online Armor is allowed to run during the tests.

    As said before before, this tests are determined as "Firewall Challenge" by the vendor.
    But on one side blacklisting the tests for AVs is forbidden and on the other side HIPS are not forced to add the tests to their whitelist.
    Or at least AV modules and HIPS modules have to be shut down during the tests for a real "Firewall Challenge".

    Cheers
     
  6. Coolio10

    Coolio10 Registered Member

    Joined:
    Sep 1, 2006
    Posts:
    1,124
    Default settings?
     
  7. ggf31416

    ggf31416 Registered Member

    Joined:
    Aug 20, 2006
    Posts:
    314
    Location:
    Uruguay
    Firewalls and HIPS will react to both known and unknow malware, unlike AVs which only reacts again known (or detected by heuristic) malware. Also testing with the AV enabled would be meaningless as every vendor would add the tests to the blacklist and get 100% in 5 seconds, without increasing protection against real malware using those techniques.

    Nowadays most firewalls are not pure firewalls but have some HIPS components. If you disable the HIPS in a firewall you should disable almost every feature in the rest, because they are also HIPS. Almost every leaktest would fail if you disable these features.
     
    Last edited: May 7, 2008
  8. Dwarden

    Dwarden Registered Member

    Joined:
    Apr 11, 2003
    Posts:
    176
    Location:
    Czech Republic
    that's described in detail on Matousec website and was discussed to death on this and comodo forums x times etc...
     
  9. subset

    subset Registered Member

    Joined:
    Nov 17, 2007
    Posts:
    825
    Location:
    Austria
    Does this mean these tests are meaningful because AVs are forbidden o_O
    Either they test programs like they are or they test just firewalls only.

    Now this are just crude tests, far from reality because of their strange settings and without any value for users.

    Reality would perhaps look like this (again for example Avira and OA):
    - Avira's heuristic might detect most of the files as HEUR/Malware, TR/Hijacker.Gen or TR/Proxy.Gen etc. >>> user is mostly well-protected.
    - Online Armor HIPS might alarm the user with multiple popups about bad things going on. >>> user is mostly well-protected.

    So my conclusion is, these tests are pointless because the methology is apparently made between the Ivory Tower and the Temple of Mammon.

    Cheers
     
  10. ggf31416

    ggf31416 Registered Member

    Joined:
    Aug 20, 2006
    Posts:
    314
    Location:
    Uruguay
    1) Unlike a firewall or HIPS, if the AV (through signatures or heuristics) detects the leaktest it doesn't mean that will detect all malware that uses the same method
    2) Testing AVs is quite different than testing HIPS or firewalls. Also there are other well-known AV testing organizations.
    3) HIPS are much closer to firewalls than AVs to firewalls. An Antivirus analyses the code of the programs and is based mainly on blacklisting. Firewalls and HIPS monitor the behaviour of the program and are based on whitelisting.
     
  11. The_1337

    The_1337 Registered Member

    Joined:
    Aug 10, 2007
    Posts:
    112
    hips isnt a firewall either so i dont see why that would be included. i mean it's called a firewall challenge not firewall and hips challenge.
     
  12. Einsturzende

    Einsturzende Registered Member

    Joined:
    Apr 14, 2008
    Posts:
    390
    Location:
    neubauten
    But personal firewall isn't complete if cannot distinguish which process made net request, so some technique similar to HIPS must be implemented (and tested of course)...
     
  13. Zombini

    Zombini Registered Member

    Joined:
    Jul 11, 2006
    Posts:
    469
    Matousec is a waste of time.. ignore them. The only products that get a 100% percent on those tests are products that are not usable in the real world.
     
  14. Dwarden

    Dwarden Registered Member

    Joined:
    Apr 11, 2003
    Posts:
    176
    Location:
    Czech Republic
    write better tests and share source ... same like Matousec does
     
  15. dawgg

    dawgg Registered Member

    Joined:
    Jun 18, 2006
    Posts:
    817
    Uhhhhhh... yes they are usable in the real world
     
  16. JanPoko

    JanPoko Registered Member

    Joined:
    May 9, 2008
    Posts:
    2
    Comodo and Online Armor are not usable in the real world?
    Thanks for that info, until now, I was convinced opposite.:rolleyes:

    And using both of them (comodo first, oa now) to my complete satisfaction. And based upon my long term experience calling both of them really the best firewalls available !!!
     
  17. subset

    subset Registered Member

    Joined:
    Nov 17, 2007
    Posts:
    825
    Location:
    Austria
    Hi,

    kindly note that using Comodo FP or Filseclab PF or whatever does not make Matousec's tests and methology meaningful or meaningless per se.

    However, the only noticeable result from this "Firewall(?) Challenge" is strangewise:
    You can estimate, that an application has some sort of HIPS(!) features or not.

    You can not even estimate, that an application has a Firewall included (ProSecurity...).

    Blank refusal, try again, fail better.

    Cheers
     
  18. Pedro

    Pedro Registered Member

    Joined:
    Nov 2, 2006
    Posts:
    3,502
    I think you'll find that most people, regardless of their opinion on Matousec, are tired of discussing the whole HIPS or FW, or whatever.

    Now he comes with some tests that should interest everyone:
    So instead of pushing the same arguments back and forth, we could try these?

    Cheers
     
  19. Einsturzende

    Einsturzende Registered Member

    Joined:
    Apr 14, 2008
    Posts:
    390
    Location:
    neubauten
    I think "Accessing Network" thingy in it is good enough for testing it like an "firewall" and with tests arsenal Matousec uses for testing FWs.
     
  20. Zombini

    Zombini Registered Member

    Joined:
    Jul 11, 2006
    Posts:
    469
    Maaaaybee by a small fraction of computer users. I can bet most people even on this newsgroup dont even know what an IP address is, so even the simplest alerts that say "IP address" are totally meaningless to 99.999% of users. Comodo and others ofcourse take the meaningless alerts to a whole new level.
     
  21. Einsturzende

    Einsturzende Registered Member

    Joined:
    Apr 14, 2008
    Posts:
    390
    Location:
    neubauten
    On the contrary...
     
  22. Diver

    Diver Registered Member

    Joined:
    Feb 6, 2005
    Posts:
    1,444
    Location:
    Deep Underwater
    SockSniff, sounds nasty.

    I have always felt the whole concept of detecting malware by waiting for it to phone home is nonsense.

    What is needed are ways to block malware installation in the first place, or at least detect its presence without relying on signatures. Products like Threatfire are a step in the right direction, but are probably not all that reliable yet.
     
  23. dawgg

    dawgg Registered Member

    Joined:
    Jun 18, 2006
    Posts:
    817
    99.99999& of users is a over exaggeration. Yes, a vast numbers of users don't know about IP addresses, but the firewalls are still usable by people. Even if they do not know what an IP address is, they can still use it by using an Allow All option rather than just allowing the IP address.

    I wouldn't say the software which scores well on leaktests are "not usable".

    The thing is, at the end of the day, with the present technology, its still possible for malware to be installed in the first place and leak data or automatically download data, and a leak proof firewall is a method to plug this hole for many users.
     
  24. Bunkhouse Buck

    Bunkhouse Buck Registered Member

    Joined:
    May 29, 2007
    Posts:
    1,056
    Location:
    Las Vegas
    It is nonsense. Once they are in your house, they can figure out a way to steal your stuff. The entire anti-leak concept is fallacious in my view.
     
  25. Mrkvonic

    Mrkvonic Linux Systems Expert

    Joined:
    May 9, 2005
    Posts:
    8,698
    Hello,

    Like I've said a kabullion times, leaktests only show how easily or in how many different ways the operating system can be fooled. Since the firewall is installed on top of the kernel, it seems like a self-defeat logic to try to control that kernel. After all, the firewall will do only whatever the kernel decides to let it see and process.

    Unless the firewall becomes some sort of super service for the kernel, which is absurd. You might as well try a different kernel.

    Furthermore, leaktests are ineffective, because:
    - People test them deliberately, knowing what to expect. It's easy blocking something called thermite or whatever, but what about something like internet explorer or explorer or svchost.
    - They assume you have been infected, which is the worst thing you can possibly do; like drinking poison and then testing if your liver can take it.

    Finally, a very good piece of malware will subvert the kernel, change the tcip stack etc - you will get precisely 0 prompts from your firewall.

    Mrk
     
Loading...
Thread Status:
Not open for further replies.