Matousec Firewall Challenge = new Test

To accept or not to accept is a personal choice. I saw some people cried "great" when they liked score and then cried "crap" when they disliked score.
As for me this is not a score that matters, but ability to control different events. If you understand what does every test do then you are able to understand how advanced your soft is and what challenges it is ready to resist. If you are looking just for score then you hardly understand anything and all those tests are just a crap for you.

 

You have the only three options.

1.) to accept the thing as they are visible - 2 mistakes that were fixed pretty fast.

2.) to believe in the conspiracy.

3.) to believe in the mystic

 

Why not to go to the site ?
===
Published: 2006/10/31

Tested products

The method how we chose personal firewalls for this phase of our project is described on the methodology page. Exact versions of tested products were:
ZoneAlarm Pro 6.1.744.001
Sunbelt Kerio Personal Firewall 4.3.246
Norton Personal Firewall 2006 version 9.1.0.33
BlackICE PC Protection 3.6.cpj
Outpost Firewall PRO 4.0 (964.582.059)

All products were tested either with a full licence that we received from the vendor for our analysis or as trial versions without any limitations in their functionality.
===

 

Is it true? I don't see any change in the "Results and comments" page.

Anyway, even if it's true, their case is quite a different from yours, though.
You are claiming that your product won 1st place and is the only firewall that got a perfect score in the test with the fixed (cheated) results - which is very unfair!
So, they want to prove that they can do the same under the same conditions.

BTW. I think OA fans and Comodo fans are the same.
They trust Matousec's test while their favorite fillwall in the ranking. OTOH, when their firewall slips off No.1, they first pretend not to care or don't trust the test results and end up bashing them! Look at how their attitude to Matousec changed before and after 3/25.

 

This is getting ugly. Can we at least keep it
on an adult level and be courteous to one
another and cease with the namecalling ?

 

This is not a secret:

http://forums.comodo.com/feedbackco...esults_2008-t20986.0.html;msg146625#msg146625
===
« Reply #68 on: March 27, 2008, 05:07:09 PM »
^^^^^^^^^^^^^^^^^^^
[skip]

We asked Matousec to re-test and, as per their rule, will pay accordingly.

[skip]

Melih
===

But as you know vendor is free to publish or not to publish the results of paid testing. May be something went wrong ?

 

I call the developer of OA a cheater because he fixed the test results with taking the paid (re)test, in which they knew what was gone be tested. (- breaking rules, creating an unfair advantage) Plus, his website now claims:

or

based on the fixed results, --- which is absolutely unfair and dishonest (- lying, deception, fraud, trickery, imposture, imposition).

Besides, he criticized us saying:

But, the fact is, it is He who is obsessed with getting No.1 spot in Matousec's Rankings and does anything for that.

Also, since I learned that their guy would hold a QA session on OA's forum, I have doubted Matousec has some sort of association with the developer of OA. And remember, before Matousec crowned it No. 1, OA was a mediocre firewall at best (In case you don't know that, check this out: http://www.techsupportalert.com/security_HIPS.htm).
I smell something fishy...

BTAIM, I don't understand why you did't see what I mentioned in the last post fits into the definitions, Pete. I guess it's because you're bias for OA. Since you are a moderator, you should be more objective, neutral and nonjudgmental.

 

Matousec tests firewalls against specific techniques which can be used by malware to bypass or terminate them (levels 1-8 ) and the stability of the drivers (levels 9-10). The score is calculated using the matousec's methodology and reflects the ability of the specific version of the firewall to control these techniques under the test's conditions, not more, not less.
The score shouldn't be used as an indication of the overall firewall's quality as it doesn't include other important aspects such as inbound protection, usability, GUI, logging, price, support neither "additional features" as Anti-spam, Anti-phising, pup-up blocker, etc.

 

Your link shows the data related to 2006 year. And nobody objects that back in 2006 OA was far from perfect HIPS. But time goes. 2007 summer OA introduced v2. The first Matousec test rated OA as very good and it was 7-th place. Then OA moved to 3-d place, then OA moved to 1-st place. Then, with the new tests OA degraded to 2-nd, but only for a week. All the leaks were addressed quickly and retest submitted in a full accordance to the Matousec rules. The same thing now does Comodo, they submitted paid retesting to achieve excellent score (great of them).

So, what strange or unfair do you see in the fact that software is been developing ? Do you like the picture when there is only one program forever and for everybody ?

Edit: as for me, I'm happy as a user there is such a competition. This helps to all of us independently of what product do we use. I think the World is big enough for everybody who does his best to have a success.

 

But maybe we have been mistaking the whole time.

That was a post from ailef the online armour tester. Did you take notice of the version he tested? There is also more questions being asked if a firewall cannot pass a test 100% it should count as a fail. And Yes, guess which firewall that is......

 

To all:

First of all, it's about time to stop the gratuitous, completely unsupported, and barely inferential extrapolations of reality that we've seen all too much of in this thread. Deal with public and known facts, not idle speculation.

With respect to moderation - if you have an issue, address it via PM. Do not pull the discussion off course in mid-thread with idle commentary of moderation.

If you cannot adhere to these guidelines, this thread will be closed without further warning or comment.

Blue

 

About the Matousec test, I am starting to think that allowing paid re-tests is wrong and creates results that can mislead consumers. Say you take an algebra test, and you miss one out of ten questions and get a 90%. The teacher gives it back to you with corrections and you say, "damn, I forgot to carry the 1 and my answer came out wrong." Then your teacher says "Hey if you pay me $20 I'll let you do that question over." So you give him $20 and now you have a 100%. There were 3 other kids in your class that also got 100%. Are you as good as those kids?

Say instead, you are in another algebra class and you are looking for a tutor and you heard that there were four kids who got 100% on the last test. If someone told you that one kid paid the teacher to redo the test and raise his grade, would that affect who you would choose as a tutor? I believe it would, which is why I am uncomfortable with paid re-testing.

So following this logic, I don't think Comodo should have been allowed to pay for a re-test, but Matousec allows it so w/e. As for Online Armor, from what I gather it only failed because while it did detect and create a alert for the actions of the leak test, it automatically defaults to allow the action if there is no user input. Whether or not this can be considered a genuine failure should have been for Matousec to decide and adjust the score accordingly with OA making it's case. But again, the paid re-test was available.

 

Actually, the appropriate education based analogy is you take a full Algrbra course and don't do as well as you desired, so you reregister, pay the course fee again, and the cards fall as they will the second time around. You may do better, you may do worse. You pay the fee again since you taking up someone's time and effort. By the way, this is done in real life everyday.

As always, caveat emptor.

Blue

 

OA failed because of bugs in its drivers. The level 9 test, Driver Verifier, is a microsoft stability test that must be passed to obtain the "windows logo". The timeout problem did not affect the score, but it could be a problem if the user keep the alert unanswered while looking for information about it.

 

I'm not even sure this is a bug, because execution alert acted just opposite -- in case there was not any user input execution was blocked. Looks like "by design". In most cases should work OK because most programs starts to act right after the start. But .. but in case malicious program would wate until a user is out and then start to act, then there could be bad outcome, of course.

 

I didn't call the timeout problem a bug. The bug is that every test from Microsoft's testing tool caused a system crash.

http://www.matousec.com/projects/firewall-challenge/level.php?num=9
http://www.microsoft.com/whdc/DevTools/tools/DrvVerifier.mspx

 

Agree

And to avoid oneliner .. On my system MS verifier even didn't produce a BSOD, it was just a reboot before login process. This behavior was the same for all the verifier options. Seems like mistake in the DriverEntry. Most probaly not verified memory allocation.

 

My sincere congrats to Comodo users

http://www.matousec.com/projects/firewall-challenge/results.php

They did it. I hope this will bring a peace here

 

With that comment, let's bring this discussion to a close. Specific technically oriented discussions can be started in separate and technically focused threads.

Blue