New Matousec firewall test, we got new leader...

[czullo]

Check this out:

http://www.matousec.com/projects/firewall-challenge/results.php

Congrats Comodo

[Peter2150]

Removed a post that was an unsubstantiated allegation. Please no posts of that nature.

Pete

[Rain_Train]

Ah, and they continue to test Mamutu... :thumbd: Maybe ClamAV will be next :dry: .

In any case, thanks for sharing the link.

[BrendanK.]

They didn't make a new test, they just retested Comodo :ouch:

[raven211]

-{ Quote: "Comodo is amazingly light, doesn't slow down startup at all, and is very powerful. Congratulations. A truly amazing product...and absolutely free too. Just use it and go have a few lunches out, rather than pay for security. :P" }-

... OR you could configure your hardware Firewall correctly and don't give a crap of either these leak-tests or the hassle of software FWs - then concentrate on the malware which is the real issue left. Simply run a free setup or buy products which are worth their price - AntiVir Professional or Premium to name two examples with Premium going for 19 euros. Compare that to Symantec's NAV offering which I normally prefer for many reasons at 39$. That made me atleast make a temporary switch...

[Iam_me]

No news really. Comodo technically has been #1 there "almost" since the lunch of the 84 set of tests.. ::) :)

Anyway now it says so on the paper also. :thumb:
Comodo - keeps users protected. :argh:

[raven211]

HIPS and Sandboxing techniques, most often hands-down, but I'm personally more than happy using behaviour blocking and strong heuristics together with (generic) signatures.

[Peter2150]

-{ Quote: "No news really. Comodo technically has been #1 there "almost" since the lunch of the 84 set of tests.. ::) :)

Anyway now it says so on the paper also. :thumb:
Comodo - keeps users protected. :argh:" }-

Not only "No news really", but nothing really significant, other than fodder for the fan boy's on either side. No. 1 will switch back and forth based on version, testing time, and maybe even the phase of the moon.

In terms of the protection offered to 99% of the users, all the top products will do the job equally well.

Pete

[Tunerz]

I think the software leading the tests are only as good as the users behind it. No matter how it passes PoC tests, it would all fail if the user doesn't know much about what they are using at all.

[twl845]

Wait until OA launches their new version which is now an RC version. The test results may be different. :isay:

[Rednose!]

-{ Quote: "Wait until OA launches their new version which is now an RC version. The test results may be different. :isay:" }-

Does it matter ???

-{ Quote: "Not only "No news really", but nothing really significant, other than fodder for the fan boy's on either side. No. 1 will switch back and forth based on version, testing time, and maybe even the phase of the moon.

In terms of the protection offered to 99% of the users, all the top products will do the job equally well.

Pete" }-

You are perfectly right here :)

Greetz, Red. ( A Comodo fan MAN ;) )

[twl845]

-{ Quote: "Does it matter ???




Greetz, Red. ( A Comodo fan MAN ;) )" }-
It only matters to the folks who follow this stuff. I have used Comodo, and now OA, and they're both great. The difference for me is the complexity (for a dummy like me) of Comodo, and the simplicity of OA which I can handle. ;D What I was saying was that OA's last version, compared to Comodo might put Comodo in 1st place, but OA has been developing the next version for a few months and has released the RC version to try. If the RC version is a good indication, the 1st place (not that it matters) may shift when OA releases their final copy.

[Iam_me]

-{ Quote: "It only matters to the folks who follow this stuff. I have used Comodo, and now OA, and they're both great. The difference for me is the complexity (for a dummy like me) of Comodo, and the simplicity of OA which I can handle. ;D What I was saying was that OA's last version, compared to Comodo might put Comodo in 1st place, but OA has been developing the next version for a few months and has released the RC version to try. If the RC version is a good indication, the 1st place (not that it matters) may shift when OA releases their final copy." }-


It might..

Comodo has decided to not pass all tests since it sees some tests as irrelevant.
https://forums.comodo.com/empty-t30896.0.html;msg222603#msg222603

OA and Outpost just pass tests to look good on paper.. They don't care if the tests are unnecessary to pass. :-\ :-\
You can always intercept stuff. But as long as it can't be used to do something reasonable by a hacker then its unnecessary interception.

Also that CIS was #1 is no news really since it has done so for months. Just not on paper, but technically since some bugs was fixed.
(A fixed version that dealt with the tests was released days after 3.5 was originally tested. And CIS was since then technically #1...)

[Rednose!]

Boys, boys, boys ...

It is only a leak test ;)

Greetz, Red.

[yashau]

Bah I wish the guys over at the Comodo forums were as nice as their software.

[m00nbl00d]

So, only COMODO was retested? Why?

I never really understood why this test is called "Firewall Challenge". No firewall is being, practically, tested.

What they are testing is a functionality that began to be part of firewalls. Sure, every firewall should have a nice outbound traffic control. But, that's it.

And, what is being tested is not the capability of a firewall to block unsolicited inbound and outbound traffic. All Matousec tests is a capability to block "leaks", and mostly to what happens within the system, and not what goes out or comes in.

I'd like to see a test performed to show the most efficient firewall preventing, in first place, inbound traffic. But, then I guess everyone would realize Windows own firewall is more than fine for that task.

Heck, can't Matousec just test the capability of a firewall to prevent hackers from getting into a system?

Then, I'd like to see a test done for outbound traffic control, which, Windows own firewall is not that easy to work with (The Windows advanced firewall.).

[Iam_me]

-{ Quote: "So, only COMODO was retested? Why?
" }-

Not true.. The following was retested:

* BitDefender Internet Security 2009 12.0.12.0
* Comodo Internet Security 3.8.65951.477
* ESET Smart Security 4.0.417.0
* ZoneAlarm Free Firewall 8.0.298.000

However Many others has already been retested.. Among them Outpost..
And If CIS was retested earlier wouldn't have mattered.. Since it has passed those tests for months..:wacko: :thumb:

Not that it matter but it was a free retesting. (all get the right to that each 3 month I think..)

[czullo]

Eset smart security gets +1% from 3 to 4 ;)

[sded]

I think it's great that there are so many products out there now with good antileak properties. And no one seems to be targeting the firewall test specifically, since the tests are all available for anyone to run, and if you want to distort your product just to pass the tests you certainly can get 100% . Congratulations to all, and to Matousec for bringing it all out into the open. :)

[andyman35]

-{ Quote: "Bah I wish the guys over at the Comodo forums were as nice as their software." }-
What do you mean? I'm the nicest person I know :P

[Peter2150]

-{ Quote: "I
OA and Outpost just pass tests to look good on paper.. They don't care if the tests are unnecessary to pass. :-\ :-\
" }-

And of course you have documented evidence to support this allegation.

[alex_s]

-{ Quote: "IOA and Outpost just pass tests to look good on paper.. They don't care if the tests are unnecessary to pass. :-\ :-\

" }-

This is nonsense. They perform not on the paper, but on the real tests. Every test demonstraits some security hole, they just do not think that security hole may be unnesesary to be handled.

-{ Quote: "

You can always intercept stuff. But as long as it can't be used to do something reasonable by a hacker then its unnecessary interception.

" }-

Hm.

Kill3f - failed.

"Comodo Internet Security failed Kill3f test because process “cssurf.exe” that is installed with the product could be
terminated under rare circumstances by this test even if the user denies all queries of the product that alert about the attack against this process."

Which does mean Comodo process protection is flawed.

SSS - failed.

"Unwanted user logout was not prevented."

SockSniff - failed.

Which does mean Comodo cannot control Raw Sockets. This test is from "keyloggers" type, but instead of keyboard it invisibly sniffs your network traffic (including sensitive information).

Crash7 - failed.

Which does mean Comodo can be terminated by any usermode program that uses this technique.

I do not think these tests are unnesesary to pass, especially crash tests.

[Rednose!]

This has already been answered on Comodo forums, most of it by Egemen month ago ;)

http://forums.comodo.com/feedbackcommentsannouncementsnews_cis/comodo_back_on_top_at_matousec-t37571.0.html

Greetz, Red.

[Leolas]

-{ Quote: "OA and Outpost just pass tests to look good on paper.. They don't care if the tests are unnecessary to pass. :-\ :-\
You can always intercept stuff. But as long as it can't be used to do something reasonable by a hacker then its unnecessary interception." }-

if this statement was true, OA would annoy the user with thousands of popup regarding unnecessary stuffs, while Comodo would have less popup and would be much easier to use.. But a lot of people say that OA is easier to use. :lurking: There must be something wrong.

Plus, OA doesn't pass the tests just to look good on the paper. May I remind you that OA was originally an HIPS, and Comodo a firewall?

As Pete, I believe you need more documentation..

[alex_s]

-{ Quote: "This has already been answered on Comodo forums, most of it by Egemen month ago ;)

http://forums.comodo.com/feedbackcommentsannouncementsnews_cis/comodo_back_on_top_at_matousec-t37571.0.html

Greetz, Red." }-

Yep, it was answered in a traditional Comodo way "everything Comodo fails is irrelevant and intentional". I just never found this approach convincing.
They never accept their fails, they blame everybody, but themselves. I trust the vendors who accept fails and go with "yes, we failed here, but we will do our best to fix it".

[Iam_me]

Iam sorry guys..

You are absolute correct regarding OA and Outpost.. That was a silly post of me. I got nothing to back it up with, and I don't believe what I posted earlier.

::) ::)

I made a mistake and Iam sorry..

Won't happen again. Sorry.

[alex_s]

-{ Quote: "Iam sorry guys..

You are absolute correct regarding OA and Outpost.. That was a silly post of me. I got nothing to back it up with, and I don't believe what I posted earlier.

::) ::)

I made a mistake and Iam sorry..

Won't happen again. Sorry." }-

Never mind. In any case this didn't look like your own idea, this was more like Melih's idea. Take my advise, put Melih on your twit-list, this will save you from a lot of silly ideas and situations.

[Iam_me]

-{ Quote: "Never mind. In any case this didn't look like your own idea, this was more like Melih's idea. Take my advise, put Melih on your twit-list, this will save you from a lot of silly ideas and situations." }-

Maby you are partially right. I presented comodos stance on those tests and made a bad assumption about 2 companies without doing the research.

Anyway lets not get off topic entirely. Iam sorry for what I said some posts earlier I admit it shouldn't have been said. :wacko: :thumb:

[twl845]

-{ Quote: " I trust the vendors who accept fails and go with "yes, we failed here, but we will do our best to fix it"." }-
Me too. That's what I like about Mike Nash at Tall Emu, the CEO who makes OnlineArmor. :thumb:

[BrendanK.]

Guys, let's not turn this into a fan boy flame fest :)

It all depends on the user TBH. If a user get's a pop up and says "Keylogger detected", and say the program was a crack or keygen, that user may still click "Allow" no matter what it says in the leak test.

Secondly it's all up to you on what you prefer in a firewall and/or HIPS. If the firewall/HIPS suits what you want, and protects you the way you want to be protected, there should be no argument as to which firewall/HIPS is better.

Also, Online Armor has NOT been retested, and as for Comodo, I'm sure they have made a wonderful product but that is not to say there are no cons which match the pros of the firewall. And yes, the same can be said for OA and all the other firewalls.

[alex_s]

-{ Quote: "Guys, let's not turn this into a fan boy flame fest :)

It all depends on the user TBH. If a user get's a pop up and says "Keylogger detected", and say the program was a crack or keygen, that user may still click "Allow" no matter what it says in the leak test.

Secondly it's all up to you on what you prefer in a firewall and/or HIPS. If the firewall/HIPS suits what you want, and protects you the way you want to be protected, there should be no argument as to which firewall/HIPS is better.

Also, Online Armor has NOT been retested, and as for Comodo, I'm sure they have made a wonderful product but that is not to say there are no cons which match the pros of the firewall. And yes, the same can be said for OA and all the other firewalls." }-

In my tests OA public RC passes all the tests previous version failed, so theoretically it should take 100% if tested. But I think they will not release just to take the first spot on Matousec.

[opus dei]

WTF.
Is it really that important? People will choose their preferred product. Most do not even give a thought to the forums of Wilders, Matousec or Comodo

Just my 10 cents
Xui

[Kees1958]

-{ Quote: "Never mind. In any case this didn't look like your own idea, this was more like Melih's idea. Take my advise, put Melih on your twit-list, this will save you from a lot of silly ideas and situations." }-

I think his videos and post are nearly as funny as Monty Python, I am not a Comodo user but, happy to read his blog etc (and have to give Comodo credit to make available a sound FW for free).

Cheers Kees

[Kees1958]

-{ Quote: "Guys, let's not turn this into a fan boy flame fest :)

It all depends on the user TBH. If a user get's a pop up and says "Keylogger detected", and say the program was a crack or keygen, that user may still click "Allow" no matter what it says in the leak test." }-

Or use a test like I did ("through the eyes of a keylogger" PoC) and get a Trojan for free, first real infection in five years or so. It is what you say, you know, but you still click okay. :-[

[BrendanK.]

-{ Quote: "Or use a test like I did ("through the eyes of a keylogger" PoC) and get a Trojan for free, first real infection in five years or so. It is what you say, you know, but you still click okay. :-[" }-

Haha. By the way no one has said it is a trojan yet, but oh wells :) Plus it was a generic detection ;)

But yep, if you trust something (or do not think it is malware), you will most likely click OK, despite the warning, and therefore become infected (if it is malware).

[twl845]

-{ Quote: "WTF.
Is it really that important? People will choose their preferred product. Most do not even give a thought to the forums of Wilders, Matousec or Comodo

Just my 10 cents
Xui" }-
I do! Wilders is where I get my education. After only 8 posts how would you know? ???

[Rednose!]

-{ Quote: "Yep, it was answered in a traditional Comodo way "everything Comodo fails is irrelevant and intentional". I just never found this approach convincing.
They never accept their fails, they blame everybody, but themselves." }-

Please read Egemens answer and than come with arguments if you beleve he is not right. You are welcome on Comodo forums for a discussion with him :)

But what you are doing now is just trolling :-\

Greetz, Red.

[alex_s]

-{ Quote: "Please read Egemens answer and than come with arguments if you beleve he is not right. You are welcome on Comodo forums for a discussion with him :)

But what you are doing now is just trolling :-\

Greetz, Red." }-

I posted the list of the failed tests and my understanding of what can happen because of those fails. I don't think those fails are unnesesary to fix. But, Jeez, why should I go to Comodo forum ? It was said here (about unnesesary tests) and this is why I argue it here. And I would be glad to meet Egemen here. I think he is a nice guy, professional developer and interesting person, but this is his chief who spoils all the party and forces him to say what he probably doesn't want to :)

For me this is quite obvious, that failed crash test is a security hole. I don't believe anybody can argue it being professional enough and sincere enough.

The only questionable failed test is SSS, but since this is actually questionable I leave it without comments. Though, coming from the fact other products can handle it gracefully, I think they'd better did it.

[Iam_me]

-{ Quote: "
For me this is quite obvious, that failed crash test is a security hole. I don't believe anybody can argue it being professional enough and sincere enough.
" }-

Maby that can be argued.. But according to Egemen CIS still intercepts everything even after such a crash.. So security isn't really bypassed.

This is his own words about it:
-{ Quote: "
--------------
crash7.exe: This test tries to allocate all the memory of the computer to crash applications including the security software
--------------

It might be possible for an application to crash if there is no more computer memory available. This is usually a random case. We do not plan to make any changes to pass this test because

* The crash can be random, intermittent and ubiquitous
* Assuming CFP/CIS processes also crashed, there is no real threat to the system because by terminating CFP/CIS, malware will not gain any advantage for byapssing Defense+.

So in summary: by terminating CFP/CIS, Defense+ will not be able to be bypassed." }-

[Rednose!]

-{ Quote: "but this is his chief who spoils all the party and forces him to say what he probably doesn't want to :)" }-

Do you realy beleve that ??? You are a funny guy ;D

Greetz, Red.

[alex_s]

-{ Quote: "Maby that can be argued.. But according to Egemen CIS still intercepts everything even after such a crash.. So security isn't really bypassed.

This is his own words about it:" }-

How crashed application can do its work. And if that application did nothing security-related, why it is in the pack ? My logic is very plain - every program in a security pack is security related. If it crashes this is potential security hole.

Let us take example with FF. There was issue reported that some tricky style xml can crash FF and POTENTIALLY can cause arbitrary code execution. FF team didn't come with the long explanation that in real world the chance this bug hardly ever can be exploited. Instead they admitted the bug and came with "will be fixed as soon as possible".

Security is a field where there is no space for the words like "real world differs from tests, low chances, crash is not a hole". Professional security can only admit reported bug and immediately fix it. Any other approach kills a trust in a vendor, especially when a vendor tries to explain that "a crash is nothing to be worried about, because this is random and other parts still work". This "explanation" can work for nonprofessional users, but ANY security expert woudl make a laugh of this "explanation" in the best case and would go very angry in the worst case. A crash says there is something wrong, and since developer cannot fix it, this "something" can potentially have unpredictable outcome.

I want that you got me right. This is not a bug that matter, there is not just a single complex application all over the world that would be bugfree. This is vendor's approach that matter. The only approach for professional security is "accept and fix", any other approach is nothing but demagogy and unfair marketing.

Another example of unfair (I'd say dirty) marketing is CMF. Comodo site states this is "Ultimate" protection from BO attacks. But this is not true. For one you can easily bypass CMF just moving malicious code from heap or stack to legitimate memory before calling any API, for two DEP is much more effective because DEP intercepts not only API calls that originates from heap or stack, DEP prevents execution of such a code.

This is what drives me nutes. And which drives me nutes even more this dirty marketing strategy is very effective ! But if product quolity was the same as marketing, I'd say "Kudos, Comodo". Instead I see that it just fools the people. Gosh. I promised to myself not to jump in Comodo related topics, but this appeared to be beyond me. I just cannot stand such things, sorry.

[Julian]

Even if no Comodo processes are running all unknown operations will be blocked - there is absoulutely no security risk. And I bet if Comodo wanted to they could easily add "protection" for this test like they could do it for socksniff (U read egemens comment?).

Well, I personally don't need BO potection but it has been updated with ver. 3.9 beta. Maybe it's better now.

And alex_s, you said that Comodo wasn't able to protect other processes because of kill3f...
Does OA or any other HIPS perfectly protect other processes? The reason why CIS "failed" this test is just because it brings with it an optional process which is not even needed anymore for anything, if other HIPS or firewalls would bring optional processes they would maybe also "fail" some tests.

Don't make Comodo products as bad as Melih or some other people in their forum are...

[Peter2150]

Whoa. Time out folks. This is not going to become another bashing thread. One more of those type posts, and it's closed.

Pete

[Hipgnosis]

Can somone tell me what the memory usage is for Comodo filewall alone and with HIPS enabled? (If there is a difference). Thanks

[andyman35]

-{ Quote: "Can somone tell me what the memory usage is for Comodo filewall alone and with HIPS enabled? (If there is a difference). Thanks" }-
On my setup cfp.exe running at about 4,500k,cmdagent 2,700-4,800k. (Everything enabled)

[alex_s]

-{ Quote: "On my setup cfp.exe running at about 4,500k,cmdagent 2,700-4,800k. (Everything enabled)" }-

Ufff ... This is not memory usage, this is working set size which can be set programmatically despite of the real memory usage. Real memory usage approximately is virtual memory usage in standard task manager and "private bytes" if you use process explorer, that includes the memory that is really allocated and used by a program but is temporary pushed out to a swap. Also pagefaults and CPU time do matter. To show approximate resource usage you need to show something like this:

[Julian]

I got one question: Detects OA it if a programs want to hide something in an ADS?

Btw: You all got just 512MB ram or why concerning about it? I can't get behind this...

[alex_s]

-{ Quote: "I got one question: Detects OA it if a programs want to hide something in an ADS?

Btw: You all got just 512MB ram or why concerning about it? I can't get behind this..." }-

Yes, ADS makes no difference. I'm not sure does it stress the fact something is in ADS or not (thought it was requested), but operations with ADS are treated like the normal file operations.

[andyman35]

-{ Quote: "Ufff ... This is not memory usage, this is working set size which can be set programmatically despite of the real memory usage. Real memory usage approximately is virtual memory usage in standard task manager and "private bytes" if you use process explorer, that includes the memory that is really allocated and used by a program but is temporary pushed out to a swap. Also pagefaults and CPU time do matter. To show approximate resource usage you need to show something like this:" }-
Of course but I answered the question in the context I feel it was asked.Most average users compare products based on resource usage as shown in task manager,not random,variable usage within specialist software such as the admittedly good Process Explorer.Also throwing out terminology such as CPU time and the likes just flies over the head of most folks.

*Edit*Here's a more detailed appraisal for you.

[alex_s]

-{ Quote: "Of course but I answered the question in the context I feel it was asked.Most average users compare products based on resource usage as shown in task manager,not random,variable usage within specialist software such as the admittedly good Process Explorer.Also throwing out terminology such as CPU time and the likes just flies over the head of most folks." }-

Yep, most users compare the resources coming from taskmanager default picture, but this can be very misleading. When I tested Comodo it took over 50MB of RAM and in TaskManager it showed very low memory usage. I think people want to know real memory usage, not fake. It is possible to set 1 MB working set for a program that really takes 100MB.

SetProcessWorkingSetSize(GetCurrentProcess(), 1024 * 1024, 1024 * 1024);

This call will make ANY program to use 1MB of memory in TM.

[andyman35]

-{ Quote: "Yep, most users compare the resources coming from taskmanager default picture, but this can be very misleading. When I tested Comodo it took over 50MB of RAM and in TaskManager it showed very low memory usage. I think people want to know real memory usage, not fake. It is possible to set 1 MB working set for a program that really takes 100MB.

SetProcessWorkingSetSize(GetCurrentProcess(), 1024 * 1024, 1024 * 1024);

This call will make ANY program to use 1MB of memory in TM." }-
You're correct that task manager only gives a partial picture in regards to resource usage of course,so for the fuller picture I've edited my post.

[Peter2150]

Guy's this thread is now all over the place. It's about the Matousec test and that's it. It is not another discussion about different product resources. Start another thread for that. Since it's apparent there's not much more to say about the test, i think the time has come.

Closed.