credential of matousec's tests ?

Discussion in 'other firewalls' started by coldplay, Apr 17, 2007.

Thread Status:
Not open for further replies.
  1. coldplay

    coldplay Registered Member

    Joined:
    Nov 12, 2006
    Posts:
    191
  2. Seer

    Seer Registered Member

    Joined:
    Feb 12, 2007
    Posts:
    2,068
    Location:
    Serbia
    Hello :)

    Are you concerned about the credibility of Matousec as a company and their abilities to perform such tests? These are not firewall tests, it's only comparison of outbound features in different firewalls, and should not be used in overall evaluation of a firewall. As you may have already noticed, this new table includes some non-firewall applications as well... Leak-proofing is not the primary function of a firewall, so please do not take this page as your sole reference. You may also find out that the top rated firewall on that page lags in some other features... which is mentioned on the same site elsewhere.

    Cheers. ;)
     
  3. aigle

    aigle Registered Member

    Joined:
    Dec 14, 2005
    Posts:
    11,164
    Location:
    UK / Pakistan
    U are right.
    I want to know about a FWs capability besides leaktets. Where I can find? Are all of them same in this regard?
     
  4. Rmus

    Rmus Exploit Analyst

    Joined:
    Mar 16, 2005
    Posts:
    4,020
    Location:
    California
    Hi Aigle,

    Before testing a firewall's capability, you have to define for yourself what a firewall is. Or that is to say, what you want the firewall to protect against.

    The classic definition of a firewall is to monitor inbound attempts to connect. Once you have your firewall rules configured to permit the needed traffic through designated ports, it is easy to check to see if all other ports are closed.

    There are port scans at grc.com and Sygate, for example, which will let you see instantly how your firewall reacts to the probes.


    regards,

    -rich

    ________________________________________________________________
    "Talking About Security Can Lead To Anxiety, Panic, And Dread...
    Or Cool Assessments, Common Sense And Practical Planning..."
    --Bruce Schneier​
     
  5. TairikuOkami

    TairikuOkami Registered Member

    Joined:
    Oct 10, 2005
    Posts:
    3,418
    Location:
    Slovakia
    True, I would also like to see some nice comparison chart with details like:
    Realtime and logging monitoring, rules creation, advanced settings and ect.
    Though, if a firewall blocks leaktests, it is might be good at other things too.
    Because those tests test firewall's engine, highest possible settings and so on.
     
  6. aigle

    aigle Registered Member

    Joined:
    Dec 14, 2005
    Posts:
    11,164
    Location:
    UK / Pakistan
    Thanks but unfrtunately I can,t test it as I use a proxy server on dial up.
     
  7. Seer

    Seer Registered Member

    Joined:
    Feb 12, 2007
    Posts:
    2,068
    Location:
    Serbia
    Hello, fellow members. :D

    As I can see, Matousec are taking their tests very seriously, and the results presented there for public insight are just the tip of the iceberg. Detailed test results are tagged with a price for vendors who know exactly how to interpret them. This is a good approach, as I have noticed that the users are taking these tests perhaps too light-heartedly and tend to look at the software features only, overlooking the flaws found. I am certainly no expert here to judge the credibility of Matousec's efforts but I have already mentioned before that I don't find Matousec site and advisories they publish aimed at the average users. You will notice that after each single review of a firewall, they also give a listing of possible bugs. This is, I believe, the main point of Matousec reviews. If they think that the found bug is critical and easily discoverable for a given firewall, they will publish an advisory (warning) for it. Other review results, such as firewall features seen in those tables, can be easily checked by users themselves. As Rmus already mentioned, there are a few online tests which will check your inbound protection with a few clicks of a mouse. Also, you can download a few tests which will try to connect in order to tell you how good your outbound is.
    Actually, Matousec are presenting us with a table containing overall features here, but the table is referring only to a few products and as I see, it is pretty much simplified. For example, you have only a check-mark for 'inbound connection control' without further explanation. Matousec are assuming that the possible readers possess the neccessary knowledge to do those basic checks.
    However, this table also shouldn't be used as a reference when choosing your firewall. The best firewall is the one you are comfortable with. What does the mightiest software do for you if you don't know how to properly set it up? You might well be left without defenses at all...

    Cheers :)
     
  8. Jo Ann

    Jo Ann Registered Member

    Joined:
    Jan 6, 2007
    Posts:
    619
    Could you please point me to those outbound-tests?
     
  9. Seer

    Seer Registered Member

    Joined:
    Feb 12, 2007
    Posts:
    2,068
    Location:
    Serbia
    Hello Jo Ann. :)

    The one from GRC is here (the green 'download here' button)

    The other one is pcAudit, here is a Softpedia page for download.

    PCFlank's is here.

    (have a little read on those pages about what they actually do, I believe they are all on Matousec's lists)

    I have a few more on my HD, but right now I can't find any links for them. Just wait a bit longer, I'm sure someone will post a few links more...

    Cheers. :D

    EDIT: There is Comodo's leak-test, it is called CPIL, also a keylogger test called AKLT (Anti-Keylogger Test). Go googling for a while, I beleieve you will find them in no time...
     
    Last edited: Apr 19, 2007
  10. Jo Ann

    Jo Ann Registered Member

    Joined:
    Jan 6, 2007
    Posts:
    619
    Thank you Seer... Since this thread questions the credibility of Matousec tests, as a Comodo user I feel good about the test results. But since quite a few highly-regarded FWs (eg., Sunbelt Kerio, Look 'n Stop, Sygate, ZoneAlarm Free, etc.) received Poor (or even worse) scores, it does make me wonder how these tests can be meaningful. :doubt:
     
  11. Mrkvonic

    Mrkvonic Linux Systems Expert

    Joined:
    May 9, 2005
    Posts:
    10,215
    Hello,
    Because these tests do not test firewalls. They test Windows' ability to trick itself in a thousand ways. It's asking what happens once you swallow 10gr of thalium.
    Mrk
     
  12. Jo Ann

    Jo Ann Registered Member

    Joined:
    Jan 6, 2007
    Posts:
    619
    Sorry Mrk, but would you please explain that (re... those test not testing firewalls). o_O
     
  13. Seer

    Seer Registered Member

    Joined:
    Feb 12, 2007
    Posts:
    2,068
    Location:
    Serbia
    Hello again Jo Ann. :)

    I believe they are very meaningful. But as I said, the leaktest coping ability is NOT the firewalls main purpose. The point of firewall is to stop the leaking malware to get on your machine in the first place. So, as Rmus pointed out here, the inbound protection of a firewall is much more important than outbound. The outbound protection is relying heavily on the HIPS (application control) which is incorporated in firewall. These tests only point out that Look'n'Stop's or Sygate's HIPS is somewhat weaker than those in Comodo or Jetico i.e. There are dedicated HIPS appplications with outbound control (such as SSM full) that you can install and thus make so-called 'layered defense'. There was a thread a month or two ago where I replied to your question about making a layered defense (if you remember). I haven't changed my opinion since then, and I still tend to think that the separation of defenses is the best way to go.

    This is a very important statement. I was always a supporter of theory that the firewall should do packet (network) filtering only. The thing is that the term 'firewall' is currently changing it's meaning, and almost all vendors now incorporate some kind of HIPS with their packet filter. As this is some kind of trend now, Matousec are just trying to investigate how is that synergy done. This, of course, doesn't mean that the CHX-I (a firewall with no HIPS at all) is a bad piece of software, only that it needs companions to be able to pass all those leaktests. Mrk is a great suppporter of Sygate, and I agree with him that this is one of the best firewalls ever produced. But, if you want your system to pass those leaktests when using Sygate, you will have to accompany it with some decent HIPS. In the end, I wouldn't bother much with the leaktests and I would concentrate my efforts in making a good inbound defense.

    Regards.
     
  14. wat0114

    wat0114 Guest

    If that is the only criteria considered important, then isn't Windows IC firewall all you need?
     
  15. Seer

    Seer Registered Member

    Joined:
    Feb 12, 2007
    Posts:
    2,068
    Location:
    Serbia
    Hello wat0114. :)

    Yes, it is. :D Windows firewall has excellent inbound control. I have been using it for years without a single issue. But I didn't say that the inbound should be the only users' concern. It is a must, while outbound is not. I personally don't bother much with outbound, as I try to prevent malware to get on my system in the first place (I am now repeating my previous statements here). But, to each it's own...

    Cheers. :D
     
  16. ggf31416

    ggf31416 Registered Member

    Joined:
    Aug 20, 2006
    Posts:
    314
    Location:
    Uruguay
    Firewalls are supposed to control inbound and outbound connections, and part of that outbound control is don't allow programs to connect like the firewall didn't exist. :eek:
    How important are leaktests results depend on how important is the outbound protection for you, and there are other factors to take into consideration for choosing a firewall.
     
  17. pvsurfer

    pvsurfer Registered Member

    Joined:
    Sep 1, 2004
    Posts:
    1,617
    Location:
    USA
    I am nowhere close to being a firewall expert, but it seems to me it's very presumptuous to say (or imply) that a firewall's principal responsibility is Inbound protection and not Outbound protection. Where is that written?

    Many of us have hardware-based Inbound protection (via a router), so our primary firewall requirement is Outbound protection. While I agree that some HIPS can serve that purpose, there are times (e.g., when traveling with a laptop) when a fully-featured firewall is the better choice of these two types of security tools.

    Just my 2 pennies worth. ;)
     
  18. Seer

    Seer Registered Member

    Joined:
    Feb 12, 2007
    Posts:
    2,068
    Location:
    Serbia
    Hello guys. :)

    Correct, ggf31416. In other words, not everybody needs to drive a BMW. Me, I'm perfectly happy with my SEAT Leon. :D All I need is a little care and attention when driving... The human factor (brain utilization) is the most important in driving as is in computer security.

    pvsurfer, a firewall is a packet filter. That has nothing to do with leak-proofing. But, it is only my point of view. Of course, your opinion may differ. ;)

    Regards.
     
  19. Pedro

    Pedro Registered Member

    Joined:
    Nov 2, 2006
    Posts:
    3,502
    Let's sepparate something here: Windows firewall doesn't control a program that allows inbound traffic (not simply incoming).
    For example, it's blind to Emule no?

    And how Netbios fits in?

    Correct me at will, for it is what i'm aiming at.

    ------------------
    For those who i lost here, incoming refers to flow of traffic.
    Inbound refers to traffic iniciated from outside. In contrast, outbound is everything that is started from us. Like browsing (requesting a website).
    (no i'm not an expert, my Q's alone give you that hint:D )
     
  20. pvsurfer

    pvsurfer Registered Member

    Joined:
    Sep 1, 2004
    Posts:
    1,617
    Location:
    USA
    Opinions aside (yours and mine), here are two generally accepted definitions:

    http://en.wikipedia.org/wiki/Personal_firewall

    http://www.webopedia.com/TERM/F/firewall.html

    Both definitions suggest that inbound AND outbound protection are a firewall's purpose. ;)

    Cheers.
     
  21. Pedro

    Pedro Registered Member

    Joined:
    Nov 2, 2006
    Posts:
    3,502
    But:
    Another note would be, outbound control isn't only application control (related to firewall of course). In fact, i think it can be ignored, leaving only protocols, SPI, ports, IP's etc.
    :D
     
  22. pvsurfer

    pvsurfer Registered Member

    Joined:
    Sep 1, 2004
    Posts:
    1,617
    Location:
    USA
    Getting back to the topic and its question about Matousec's tests, I don't find his Sygate scoring to be consistent with what I've experienced over the past 2 years that I've been using Sygate Personal Firewall Pro. During that period, I have been using v5.6.3408 on my laptop and I have never found it to allow any of my applications to have internet access once I prohibited such access. Therefore, I can't believe Matousec's testing is very meaningful. o_O
     
  23. Seer

    Seer Registered Member

    Joined:
    Feb 12, 2007
    Posts:
    2,068
    Location:
    Serbia
    OK guys. I believe we got stuck in the teminology here. A firewall is a firewall whether it does in/out control or only in as Windows Firewall does. A router's firewall is still a firewall, even if it does not control outbound traffic. It is only a matter of what user needs. As I said, a term 'firewall' is now changing it's meaning. But I still like to think that firewall is there to control what user can't, and that is inbound traffic (attacks). A user alone should be able to control which application connects out. If you use trusted applications and take a little care when surfing online, you really don't need an outbound control. An inbound control is a must, outbound is not.

    Regards. :)
     
  24. Seer

    Seer Registered Member

    Joined:
    Feb 12, 2007
    Posts:
    2,068
    Location:
    Serbia
    Hello. :)

    Yes, but they are not testing legitimate applications. Rather malicious-like which fork themselves onto legitimate ones (IE) in attempt to connect out. If you prohibit access for your browser to connect out, you are perfectly leak-proof. But, you can't browse either ;)
     
  25. Rmus

    Rmus Exploit Analyst

    Joined:
    Mar 16, 2005
    Posts:
    4,020
    Location:
    California
    Some firewall history:

    http://www.informit.com/content/images/9781587053290/excerpts/1587053292sc.pdf

    http://en.wikipedia.org/wiki/Firewall_(networking)

    The article continues with the evolution/development of firewall technolgy, eventually arriving at the
    • Application Layer Firewall
    • Application Firewall
    • Personal Firewall
    and others.

    Depending on the type of firewall you have, and what you want it to do, the various firewall tests now available may or may not have any relevance.


    regards,

    -rich

    ________________________________________________________________
    "Talking About Security Can Lead To Anxiety, Panic, And Dread...
    Or Cool Assessments, Common Sense And Practical Planning..."
    --Bruce Schneier​
     
Thread Status:
Not open for further replies.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.