credential of matousec's tests ?

[coldplay]

here is the most recent firewall test , 04/15/07

http://www.matousec.com/projects/windows-personal-firewall-analysis/leak-tests-results.php

[Seer]

Hello :)

Are you concerned about the credibility of Matousec as a company and their abilities to perform such tests? These are not firewall tests, it's only comparison of outbound features in different firewalls, and should not be used in overall evaluation of a firewall. As you may have already noticed, this new table includes some non-firewall applications as well... Leak-proofing is not the primary function of a firewall, so please do not take this page as your sole reference. You may also find out that the top rated firewall on that page lags in some other features... which is mentioned on the same site elsewhere.

Cheers. ;)

[aigle]

U are right.
I want to know about a FWs capability besides leaktets. Where I can find? Are all of them same in this regard?

[Rmus]

Hi Aigle,

Before testing a firewall's capability, you have to define for yourself what a firewall is. Or that is to say, what you want the firewall to protect against.

The classic definition of a firewall is to monitor inbound attempts to connect. Once you have your firewall rules configured to permit the needed traffic through designated ports, it is easy to check to see if all other ports are closed.

There are port scans at grc.com and Sygate, for example, which will let you see instantly how your firewall reacts to the probes.


regards,

-rich

________________________________________________________________
"Talking About Security Can Lead To Anxiety, Panic, And Dread...
Or Cool Assessments, Common Sense And Practical Planning..."
--Bruce Schneier

[TOMxEU]

True, I would also like to see some nice comparison chart with details like:
Realtime and logging monitoring, rules creation, advanced settings and ect.
Though, if a firewall blocks leaktests, it is might be good at other things too.
Because those tests test firewall's engine, highest possible settings and so on.

[aigle]

-{ Quote: "There are port scans at grc.com and Sygate, for example, which will let you see instantly how your firewall reacts to the probes. " }-
Thanks but unfrtunately I can,t test it as I use a proxy server on dial up.

[Seer]

Hello, fellow members. :D

As I can see, Matousec are taking their tests very seriously, and the results presented there for public insight are just the tip of the iceberg. Detailed test results are tagged with a price for vendors who know exactly how to interpret them. This is a good approach, as I have noticed that the users are taking these tests perhaps too light-heartedly and tend to look at the software features only, overlooking the flaws found. I am certainly no expert here to judge the credibility of Matousec's efforts but I have already mentioned before that I don't find Matousec site and advisories they publish aimed at the average users. You will notice that after each single review of a firewall, they also give a listing of possible bugs. This is, I believe, the main point of Matousec reviews. If they think that the found bug is critical and easily discoverable for a given firewall, they will publish an advisory (warning) for it. Other review results, such as firewall features seen in those tables, can be easily checked by users themselves. As Rmus already mentioned, there are a few online tests which will check your inbound protection with a few clicks of a mouse. Also, you can download a few tests which will try to connect in order to tell you how good your outbound is.
Actually, Matousec are presenting us with a table containing overall features here (http://www.matousec.com/projects/windows-personal-firewall-analysis/results.php), but the table is referring only to a few products and as I see, it is pretty much simplified. For example, you have only a check-mark for 'inbound connection control' without further explanation. Matousec are assuming that the possible readers possess the neccessary knowledge to do those basic checks.
However, this table also shouldn't be used as a reference when choosing your firewall. The best firewall is the one you are comfortable with. What does the mightiest software do for you if you don't know how to properly set it up? You might well be left without defenses at all...

Cheers :)

[Jo Ann]

-{ Quote: " ....Also, you can download a few tests which will try to connect in order to tell you how good your outbound is...." }-
Could you please point me to those outbound-tests?

[Seer]

Hello Jo Ann. :)

The one from GRC is here (http://www.grc.com/lt/leaktest.htm) (the green 'download here' button)

The other one is pcAudit, here (http://www.softpedia.com/get/Security/Firewall/pcAudit-Leak-Test.shtml) is a Softpedia page for download.

PCFlank's is here (http://www.pcflank.com/pcflankleaktest.htm).

(have a little read on those pages about what they actually do, I believe they are all on Matousec's lists)

I have a few more on my HD, but right now I can't find any links for them. Just wait a bit longer, I'm sure someone will post a few links more...

Cheers. :D

EDIT: There is Comodo's leak-test, it is called CPIL, also a keylogger test called AKLT (Anti-Keylogger Test). Go googling for a while, I beleieve you will find them in no time...

[Jo Ann]

Thank you Seer... Since this thread questions the credibility of Matousec tests, as a Comodo user I feel good about the test results. But since quite a few highly-regarded FWs (eg., Sunbelt Kerio, Look 'n Stop, Sygate, ZoneAlarm Free, etc.) received Poor (or even worse) scores, it does make me wonder how these tests can be meaningful. :-\

[Mrkvonic]

Hello,
Because these tests do not test firewalls. They test Windows' ability to trick itself in a thousand ways. It's asking what happens once you swallow 10gr of thalium.
Mrk

[Jo Ann]

-{ Quote: "Hello,
Because these tests do not test firewalls. They test Windows' ability to trick itself in a thousand ways. It's asking what happens once you swallow 10gr of thalium.
Mrk" }-
Sorry Mrk, but would you please explain that (re... those test not testing firewalls). ???

[Seer]

Hello again Jo Ann. :)

-{ Quote: "it does make me wonder how these tests can be meaningful. " }-

I believe they are very meaningful. But as I said, the leaktest coping ability is NOT the firewalls main purpose. The point of firewall is to stop the leaking malware to get on your machine in the first place. So, as Rmus pointed out here, the inbound protection of a firewall is much more important than outbound. The outbound protection is relying heavily on the HIPS (application control) which is incorporated in firewall. These tests only point out that Look'n'Stop's or Sygate's HIPS is somewhat weaker than those in Comodo or Jetico i.e. There are dedicated HIPS appplications with outbound control (such as SSM full) that you can install and thus make so-called 'layered defense'. There was a thread a month or two ago where I replied to your question about making a layered defense (if you remember). I haven't changed my opinion since then, and I still tend to think that the separation of defenses is the best way to go.

-{ Quote: "Before testing a firewall's capability, you have to define for yourself what a firewall is." }-

This is a very important statement. I was always a supporter of theory that the firewall should do packet (network) filtering only. The thing is that the term 'firewall' is currently changing it's meaning, and almost all vendors now incorporate some kind of HIPS with their packet filter. As this is some kind of trend now, Matousec are just trying to investigate how is that synergy done. This, of course, doesn't mean that the CHX-I (a firewall with no HIPS at all) is a bad piece of software, only that it needs companions to be able to pass all those leaktests. Mrk is a great suppporter of Sygate, and I agree with him that this is one of the best firewalls ever produced. But, if you want your system to pass those leaktests when using Sygate, you will have to accompany it with some decent HIPS. In the end, I wouldn't bother much with the leaktests and I would concentrate my efforts in making a good inbound defense.

Regards.

[wat0114]

-{ Quote: " In the end, I wouldn't bother much with the leaktests and I would concentrate my efforts in making a good inbound defense.
" }-

If that is the only criteria considered important, then isn't Windows IC firewall all you need?

[Seer]

Hello wat0114. :)

-{ Quote: "If that is the only criteria considered important, then isn't Windows IC firewall all you need?" }-

Yes, it is. :D Windows firewall has excellent inbound control. I have been using it for years without a single issue. But I didn't say that the inbound should be the only users' concern. It is a must, while outbound is not. I personally don't bother much with outbound, as I try to prevent malware to get on my system in the first place (I am now repeating my previous statements here). But, to each it's own...

Cheers. :D

[ggf31416]

Firewalls are supposed to control inbound and outbound connections, and part of that outbound control is don't allow programs to connect like the firewall didn't exist. :o
How important are leaktests results depend on how important is the outbound protection for you, and there are other factors to take into consideration for choosing a firewall.

[pvsurfer]

I am nowhere close to being a firewall expert, but it seems to me it's very presumptuous to say (or imply) that a firewall's principal responsibility is Inbound protection and not Outbound protection. Where is that written?

Many of us have hardware-based Inbound protection (via a router), so our primary firewall requirement is Outbound protection. While I agree that some HIPS can serve that purpose, there are times (e.g., when traveling with a laptop) when a fully-featured firewall is the better choice of these two types of security tools.

Just my 2 pennies worth. ;)

[Seer]

Hello guys. :)

-{ Quote: "How important are leaktests results depend on how important is the outbound protection for you" }-

Correct, ggf31416. In other words, not everybody needs to drive a BMW. Me, I'm perfectly happy with my SEAT Leon. ;D All I need is a little care and attention when driving... The human factor (brain utilization) is the most important in driving as is in computer security.

pvsurfer, a firewall is a packet filter. That has nothing to do with leak-proofing. But, it is only my point of view. Of course, your opinion may differ. ;)

Regards.

[Pedro]

Let's sepparate something here: Windows firewall doesn't control a program that allows inbound traffic (not simply incoming).
For example, it's blind to Emule no?

And how Netbios fits in?

Correct me at will, for it is what i'm aiming at.

------------------
For those who i lost here, incoming refers to flow of traffic.
Inbound refers to traffic iniciated from outside. In contrast, outbound is everything that is started from us. Like browsing (requesting a website).
(no i'm not an expert, my Q's alone give you that hint;D )

[pvsurfer]

-{ Quote: "pvsurfer, a firewall is a packet filter. That has nothing to do with leak-proofing. But, it is only my point of view. Of course, your opinion may differ. ;)
Regards." }-

Opinions aside (yours and mine), here are two generally accepted definitions:

http://en.wikipedia.org/wiki/Personal_firewall (http://en.wikipedia.org/wiki/Personal_firewall)

http://www.webopedia.com/TERM/F/firewall.html (http://www.webopedia.com/TERM/F/firewall.html)

Both definitions suggest that inbound AND outbound protection are a firewall's purpose. ;)

Cheers.

[Pedro]

But:
Another note would be, outbound control isn't only application control (related to firewall of course). In fact, i think it can be ignored, leaving only protocols, SPI, ports, IP's etc.
;D

[pvsurfer]

Getting back to the topic and its question about Matousec's tests, I don't find his Sygate scoring to be consistent with what I've experienced over the past 2 years that I've been using Sygate Personal Firewall Pro. During that period, I have been using v5.6.3408 on my laptop and I have never found it to allow any of my applications to have internet access once I prohibited such access. Therefore, I can't believe Matousec's testing is very meaningful. :wacko:

[Seer]

OK guys. I believe we got stuck in the teminology here. A firewall is a firewall whether it does in/out control or only in as Windows Firewall does. A router's firewall is still a firewall, even if it does not control outbound traffic. It is only a matter of what user needs. As I said, a term 'firewall' is now changing it's meaning. But I still like to think that firewall is there to control what user can't, and that is inbound traffic (attacks). A user alone should be able to control which application connects out. If you use trusted applications and take a little care when surfing online, you really don't need an outbound control. An inbound control is a must, outbound is not.

Regards. :)

[Seer]

Hello. :)

-{ Quote: "Therefore, I can't believe Matousec's testing is very meaningful." }-

Yes, but they are not testing legitimate applications. Rather malicious-like which fork themselves onto legitimate ones (IE) in attempt to connect out. If you prohibit access for your browser to connect out, you are perfectly leak-proof. But, you can't browse either ;)

[Rmus]

Some firewall history:

http://www.informit.com/content/images/9781587053290/excerpts/1587053292sc.pdf

-{ Quote: "Cisco IOS Software has had a type of firewall included since the early releases, in the form of packet-filtering technology. As previously discussed, this was the first generation of firewall technology." }-

http://en.wikipedia.org/wiki/Firewall_(networking)

-{ Quote: "First generation - packet filters

The first paper published on firewall technology was in 1988, when Jeff Mogul from Digital Equipment Corporation (DEC) developed filter systems known as packet filter firewalls. This fairly basic system was the first generation of what would become a highly evolved and technical internet security feature. At AT&T Bill Cheswick and Steve Bellovin were continuing their research in packet filtering and developed a working model for their own company based upon their original first generation architecture.

Packet filters act by inspecting the "packets" which represent the basic unit of data transfer between computers on the Internet. If a packet matches the packet filter's set of rules, the packet filter will drop (silently discard) the packet, or reject it (discard it, and send "error responses" to the source)." }-The article continues with the evolution/development of firewall technolgy, eventually arriving at the
Application Layer Firewall

Application Firewall

Personal Firewall

and others.

Depending on the type of firewall you have, and what you want it to do, the various firewall tests now available may or may not have any relevance.


regards,

-rich

________________________________________________________________
"Talking About Security Can Lead To Anxiety, Panic, And Dread...
Or Cool Assessments, Common Sense And Practical Planning..."
--Bruce Schneier

[pvsurfer]

-{ Quote: "Hello. :)
Yes, but they are not testing legitimate applications. Rather malicious-like which fork themselves onto legitimate ones (IE) in attempt to connect out. If you prohibit access for your browser to connect out, you are perfectly leak-proof. But, you can't browse either ;)" }-Whatever the reason, in the 2 years that I've been using it, I have found Sygate PFP to provide excellent protection and during that time I installed a large number of apps, many of which wanted to 'call home' on a regular basis. So hopefully you understand why it's difficult for me to attach real-world meaning to Matousec's tests.

[ggf31416]

-{ Quote: "Whatever the reason, in the 2 years that I've been using it, I have found Sygate PFP to provide excellent protection and during that time I installed a large number of apps, many of which wanted to 'call home' on a regular basis. So hopefully you understand why it's difficult for me to attach real-world meaning to Matousec's tests." }-

You don't need much outbound protection to control legitimate programs, as they don't attempt to bypass the firewall deliberately. You need more outbound protection if you are ultra-paranoid or if you want stronger protection against malware not detected by your AV.

[lucas1985]

Let me put some light on this rough topic. Don't forget that I'm not a security expert, so I can make some mistakes :)

The key words in this thread are: "security strategy", "firewalls" and "leaktests". If we define them at first, this will be easy to understand.

- Security strategy
There isn't a security strategy which fits all patterns of computing. Each PC user must develop his/her own security scheme which should be based on his/her knowledge, discipline, availability of resources and patience. A basic layout should include backup (which requires its own policy), a router, password management (another strategy to develop) , a antimalware app, a personal firewall and common sense/safe hex. A more sofisticated/advanced strategy may add rollback solutions, HIPS, integrity checkers, more antimalware apps, hardening, etc.
A smart, rational, well developed scheme requires a basic understanding of how malware is able to install itself into a system.
Understanding Computer Infections I (http://wiki.castlecops.com/Understanding_Computer_Infections)
Understanding Computer Infections II (http://wiki.castlecops.com/Understanding_Computer_Infections_-_Part_two)
Understanding Computer Infections III (http://wiki.castlecops.com/Understanding_Computer_Infections_-_Part_three)
When you understand how malware infects your PC, you'll be able to tackle leaktests with your security scheme. It doesn't matter if this is a job for your firewall or HIPS.

- Firewall
Firewalls are applications designed to filter network packets. Nothing more. But wait, network packets are created by applications, so a personal firewall should keep some track of the applications generating network activity. Then, we can classify personal firewalls according to their ability to control applications requiring access to network:
* Firewall with null control of applications (aka "pure" packet filters): Ghostwall and CHX-I.
* Firewalls with limited control of applications (name of executable, path, MD5 checksum): Kerio 2.1.5, Filseclab.
* Firewalls with advanced control of applications (but with varying degrees of success): all the others (LnS, Comodo, SKPF, Jetico, Blink, Outpost)
Are firewalls with null/limited control of applications weaker than the others? Absolutely no. Your security setup (in a layered approach) will take care of the "deficiencies" of the chosen firewall.

- Leaktests
Leaktests are PoC -proof of concept code- which use Windows design vulnerabilities to bypass firewall's control policy in a smart way (http://www.firewallleaktester.com/categories.htm) (i.e. no brute force attacks). Some of the leaktests techniques are already used by malware. But leaktests first need to execute. So, a execution interceptor (usually a HIPS) will block any leaktest (or malware) before it's loaded in memory. But, leaktests are usually tested against firewall, which must discern between legitimate and malicious activity of files already loaded in memory (i.e. the user has given it execution permissions).
So, who needs to pass leaktests? THE USER :) A firewall or HIPS which are "leaktest-aware" may help, but it's up to you to pass then, because when malware is executed, it's generally too late.

[Mrkvonic]

Hello,

No problem controlling outbound. It's OK. Something you even want it. Outbound control of legitimate programs.

Now comes the question what can your firewall do when subjected to malware? And this is really the question of what can malware do on Windows.

Answer: everything.

Therefore, to 100% control outbound for every possible type of code, you need total system control. This means that the only 100% firewall is one that completely patches the kernel, takes control of it and becomes the sole I/O filter for what you do.

Otherwise, there will always be some aspect of the system that you firewall will not be able to monitor.

Example: malware installs its own socket. Your firewall might not be able to monitor this at all.

The problem with leaktests is that they address an important issue that happens AFTER you infect your machine. Like I said, swallow 10gr of thalium and see how you cope.

The point is to keep malicious code off your computer and then, your worry with outbound is restricted to normal applications using normal protocols, and there you do not need any leaktests.

How to keep malicious code off your machine?

This is open to debate. Some will say HIPS, some will say anti-virus + anti-spyware, some will say anti-executables, software policies, limited user, alternative browser etc. All legitimate ways.

Choose the one best suited to your skill, understanding and needs.

If you are very paranoid and do not trust yourself too much, limited user or anti-executable is probably the best choice. If you are a geek and like to be in control, go for HIPS. If you just know what you do, firewall is enough. And so forth.

Mrk

[Peter2150]

-{ Quote: "Hello,

No problem controlling outbound. It's OK. Something you even want it. Outbound control of legitimate programs.

Now comes the question what can your firewall do when subjected to malware? And this is really the question of what can malware do on Windows.

Answer: everything.

Therefore, to 100% control outbound for every possible type of code, you need total system control. This means that the only 100% firewall is one that completely patches the kernel, takes control of it and becomes the sole I/O filter for what you do.

Otherwise, there will always be some aspect of the system that you firewall will not be able to monitor.

Example: malware installs its own socket. Your firewall might not be able to monitor this at all.

The problem with leaktests is that they address an important issue that happens AFTER you infect your machine. Like I said, swallow 10gr of thalium and see how you cope.

The point is to keep malicious code off your computer and then, your worry with outbound is restricted to normal applications using normal protocols, and there you do not need any leaktests.

How to keep malicious code off your machine?

This is open to debate. Some will say HIPS, some will say anti-virus + anti-spyware, some will say anti-executables, software policies, limited user, alternative browser etc. All legitimate ways.

Choose the one best suited to your skill, understanding and needs.

If you are very paranoid and do not trust yourself too much, limited user or anti-executable is probably the best choice. If you are a geek and like to be in control, go for HIPS. If you just know what you do, firewall is enough. And so forth.

Mrk" }-

A double Amen to this. Actually since this thread is about credentials, what the heck are his credentials. I went to the website and I couldn't tell. What turned me off even more was the "Ive found a problem with your software, and will knock you in my test but you will have to pay me to find out what" Phewy. I at least have the highest respect for GKWEB's honesty, and candor about what he does.

Pete

[ErikAlbert]

What passes through my firewall of straw, will be killed by my security softwares, if not my frozen snapshot will kill it.
So Matousec's tests don't tell me anything new, but it's good for a social chat. Matousec's tests are just another way of earning money. :)
The brilliant bad guys always manage to break any software, no matter how many times it has been tested, even by Matousec.

[wat0114]

-{ Quote: " Actually since this thread is about credentials, what the heck are his credentials. I went to the website and I couldn't tell. What turned me off even more was the "Ive found a problem with your software, and will knock you in my test but you will have to pay me to find out what" Phewy. I at least have the highest respect for GKWEB's honesty, and candor about what he does.

Pete" }-

I totally agree. gkweb's testing is done professionally and unbiased. He just gives the straight goods. Matousec wants $$ for bug reports and slams products that don't meet his standards on how a firewall should work.

[Pedro]

-{ Quote: "Hello,

No problem controlling outbound. It's OK. Something you even want it. Outbound control of legitimate programs.

Now comes the question what can your firewall do when subjected to malware? And this is really the question of what can malware do on Windows.

Answer: everything.

Therefore, to 100% control outbound for every possible type of code, you need total system control. This means that the only 100% firewall is one that completely patches the kernel, takes control of it and becomes the sole I/O filter for what you do.

Otherwise, there will always be some aspect of the system that you firewall will not be able to monitor.

Example: malware installs its own socket. Your firewall might not be able to monitor this at all.

The problem with leaktests is that they address an important issue that happens AFTER you infect your machine. Like I said, swallow 10gr of thalium and see how you cope.

The point is to keep malicious code off your computer and then, your worry with outbound is restricted to normal applications using normal protocols, and there you do not need any leaktests.

How to keep malicious code off your machine?

This is open to debate. Some will say HIPS, some will say anti-virus + anti-spyware, some will say anti-executables, software policies, limited user, alternative browser etc. All legitimate ways.

Choose the one best suited to your skill, understanding and needs.

If you are very paranoid and do not trust yourself too much, limited user or anti-executable is probably the best choice. If you are a geek and like to be in control, go for HIPS. If you just know what you do, firewall is enough. And so forth.

Mrk" }-
Excellent post Mrk. That's it. I'm tending towards that exact post.