New Firewall Tests from Leak Tester

The review is from March 2006. It's a bit of a surprise as Jetico came first, follow by Outpost and L"n"S

http://www.firewallleaktester.com/index.html

 

Looks to me like Outpost and LnS got the same score, but Outpost did edge out LnS, barely, in the leaktests. I think either one is a great firewall.

I've never tried Jetico nor paid much attention to anything about Jetico, so can't comment on its win.

*edit* On looking at the list again, if you consider the vertical line - (generic block) which means the leaktest is stopped before it can access the network, it looks like ZA Pro actually stopped more leaks than any of them by 14 to 12.

 

As always very good tests to have and to be able to reference etc, thanks gkweb.

I tried both Breakouts and Jumper on ZA Free on 98SE just to see what might happen, and they didn't even launch. As they are for XP etc it isn't surprising, but sometimes exploits can cross over, so it's nice to know i'm safe from those. I've used all your tests Apps several times in the past, and others too, and am able to block 99% of them with ZA Free in high security mode and running Winsonar.

This latest round of tests made me look into Jetico for the first time, so well done to them ! I DL the help file to read, which was very indepth and i would need more time to study it properly to set it all up correctly, if i choose to try it. But i'm not sure given my present system config, that i would gain any benefits from spending all that time doing so and installing it ? Unless somebody can explain how i may improve my security by installing it.


StevieO

 

Well it seem that bitdefender has also improved the firewall.
And specially on version internet security, there it is a button "Stealth Mode"

 

ZA doesn't get the total points for AWFT as it only stopped 8 of the 10 subtests in that test. It also got a generic block on 1 of the remaining 2 and it missed one altogether. By counting the way you are, you would be counting ZA'a generic block of one test and it's blocking 8 of 10 tests of the same quality as the checkmark for Outpost, LnS and Jetico for stopping all 10. That would obviously be unfair to elevate ZA to the same level as the 3 that did stop all 10 correctly. ZA is a good product though as it was the only firewall that managed to stop the breakout1 test. The score is accurate though as each test a firewall passed is worth one point. A generic block is worth 1/2 a point.

 

Good stuff. The only thing that does bother me is the fact the tester is heavily involved with GhostSecurity - thus putting his impartiallity at stake.

 

Where can you find a single trace of partiality when Ghostwall - or any Ghost Security product - is not included in the tests?

Cheers,
nicM

 

nicM, I think he is refering to gkweb's preference for a HIPS is AppDefend. I think he believes there is biasim since he is a beta tester of the product and also produced their procX. However, it should not really affect partiality since gkweb also beta tests jetico, DCS, and many other companies products.

Alphalutra1

 

OK, but we're not talking about the leaktest comparative anymore there - or I'm missing something?

I was reading the post I quoted in the context of theses firewalls tests (thus my reaction ) , not about the software advice section.

Cheers,
nicM

 

Nice to see that they have updated the test results, and also nice to see that ZA Pro isn´t doing that bad. Of course other HIPS should be able to fill the gap.

 

Were these firewalls tested using the programs default ruleset or rulesets created by the tester?

 

I PM'd gkweb (firewall tester), and asked him that exact question (if the tests were performed on firewalls as they are presented to the user "out of the box", or if the settings were tweaked for maximum performance). Here is the reply he sent to me:

"Yes I have tweaked the firewalls to their highest settings, every feature was enabled, everything maxed out. Global filtering rules removed if any, to be sure to be asked about any network activity.

Out of the box settings, generally and depending of the firewall, are weaker. Out of the box settings are generally purposefully not set to high to not ask too much popups to the user."

 

So basically for the average joe, the tests really mean nothing then.
Unless of course the user is a firewall guru and can create the same settings and rules as what was used for the test.

I think the test maybe should have been the other way around. Test it "out of the box" for the average joe. Give us a idea how bad the firewalls work in that way.

 

You're welcome

Well, I'm just passing along the info...even though I agree that perhaps there should be TWO sets of tests: One for the "average Joe" (which I am one of as well), and one for the most knowledgeable and advanced users.

But please....don't kill the messenger cause you don't like the message!

 

So JR where exactly did that leave you with your firewall of choice?... Since the tests really arn't directed towards the average joe and his average ruleset firewall..

 

Probably in the same position as you - UNDECIDED

Actually, I am trialing LNS on a test machine currently.....and if Blackspear's settings and "Advanced Options" being checked are "sufficiently adequate" and are what is considered to be the "optimal settings", then it is relatively easy enough to understand and properly configure. Don't know about any specific "additional rules", though.....

Hey, here's an idea for gkweb to consider for future tests, though (and to add as a disclaimer for the current round of tests, if he can remember and wouldn't mind adding this additional info):

Perhaps WHATEVER SETTINGS were used to test each firewall could be added in a "Settings and Configuration Description" section. He could have a link for each firewall, basically explaining WHICH measures and actions were taken to "tweak" each firewall to configure it to it's "maximum/optimal security setting".

Call me crazy, and I know I'm asking (or suggesting) for extra work, time and effort to be made on behalf of the tester....but I think it would help "average" (and perhaps even more advanced) users to properly set-up the firewall of their choice......

 

Hello,

"out of the box" test was done in the past, and there was two tables, one "highest settings" and one "out of the box". It has been removed since due to the big extra work it involves.

Currently there is 18 leaktests (26 real tests, AWFT has 6 tests, Wallbreaker 4 tests, etc...), 15 firewall tested, each test triple checked (really), that gives : 26 * 15 = 390 tests (triple check = 390 * 3 = 1000+ tests). If you add the "out of the box" table, you reach 2000+ tests. Given the fact that I will add other firewalls, I let you imagine the amount of work required.

I am not paid to do such tests, I do them in my spare time, and I do not have so much time.

The highest settings shows the true security potential of the firewall, I see it as a vital information. Highest settings are not so hard to do as you seem to think, basically enable every features, and disable automatic allowances.

If you want more information about "out of the box" settings, you can read more there :
http://www.firewallleaktester.com/advices.htm#01

Regards,
gkweb.

 

first of all, thanx gkweb for testing all those fw's. may I ask why Tiny isn't included in the test?

 

I would have like to have seen that as well

 

I'm not interested in you doing out of the box tests, you can forget those if you like, but you can at least tell us what the 'highest settings' you used are. It's kind of like doing an antivirus test but refusing to tell how the AV was setup, what signatures were used, what samples were used etc. It makes it hard for other people to replicate your findings if they so choose to do.

That said I agree with the following

True. I got about the same results you have with a few firewalls, so clearly, your 'highest settings' isn't some arcane art beyond all but firewall gurus.

 

Hello,

To be honest, I didn't realize that some people was wanting that much "firewall tutorial settings", and I understand your concerns.

I have added that to my "To Do" list, but honestly I don't know when I will be able to do it. If I do a page for each firewall, taking screenshots and commenting what to do, if I have 5 screenshot * 14 firewalls, I will have 70 screenshots + comments, that is a big work also, in addition to the tests.

Just for information, today I have passed my entire day on things related to my website. I have results to check with what people tell me, I am in contact with firewall vendors about current private firewalls build which should be updated next month, I am grabbing and compiling every people opinion on various forums and am preparing to modify some things on my website, etc... I have planned to do some personal things today and I didn't even have the time.

So I take seriously your request, but to achieve what you (and many other) want, either it will be asking again some time, or either I will have to request help from other people (of course checking the tutorial before validating it).
For now I cannot do it since I am working on my next website update, but be sure I will not forget it

In the meanwhile, I can suggest to anyone interested to check the official firewall forum and request there how to tighten their firewall security.

Regards,
gkweb.

EDIT : about Tiny, it is in the reward page, and so cannot be in the scoreboard. But the next website update should change this point, and I plan to test it later.

 

sorry I don't understand surely tiny warrants testing even if it was rewarded in 2004

 

After looking at the good tests by 'gkweb' I have to say that it has kinda changed my view of Nortons firewall, compared to what I used to think of it.

 

I am not surprised why Jetico becomes no. 1 in the test. I have tried and used this firewall in the past and it "bombarded" me with lots of pop-ups, I know that its just one of the normal character of the best protection software nowadays and that is to inform pc user what really is going on inside the system. Getting indepth on it this firewall was really designed for advanced users as I cannot understand clearly how it works and how to configure it but I know that it's one of the best. Still, I don't have enough patience on its machine gun like pop-ups then I have to uninstall it and never have returned on using it again.

AppDefend looks more colorful than ProcessGuard that's why gkweb choose this thing (just kidding ) Seriously, even if AppDefend was still in beta stages it already looks very promising and offers a lots of things that probably PG misses. Its Network access control was I think was one of its best features not found on PG. I was anticipating that PG will improve from its present protection capabilities and I was hoping that it'll also adds some more features not found in AppDefend.

 

If the settings could be shown for, say, the top 3 firewalls in the test, maybe that would be a more reasonable request for now?

Of course, if the firewall uses an almost constant 90 or 100% cpu (and also gives a mound of pop-ups that regular users can't understand) then who needs adware: I just installed it, myself.
(points at Outpost )

So, it would be interesting to see in the test the amount of resources used by each firewall.