Matousec's new Firewall Challenge "wrinkle"

[wat0114]

From his homepage:

www.matousec.com

-{ Quote: "Firewall Challenge recommendations (2008/04/20 15:16)

Firewall Challenge revealed several notable security products. We have decided to recommend the best products to you via affiliate programs of their vendors. Every product that scores at least 80% in the challenge and thus receives our mark of Very good or Excellent Protection level may become a recommended product. We contact the vendors of such products in order to join their affiliate programs. The vendors that agree and provide us the necessary technical instruments will be linked from the result page of Firewall Challenge.

What does it mean for end users? If you like Firewall Challenge, you can help us and support this project by buying one of the recommended products through the links in Firewalls' ratings table. To get more information about how does it work, please read How you can help us paragraph on the index page of Firewall Challenge.

We start our recommendations with affiliate programs of Agnitum Ltd. and Kaspersky Lab. If you are going to buy any of their products or prolong your license, you can help us by doing so through our links. In the next weeks we will try to join affiliate programs of vendors of other great products tested in Firewall Challenge." }-

Btw, Credit for pointing this out goes to Outpost Firewall Support Forum member/moderator, minoka :)

[Smokey]

Good to know, over and out for Matousec and his credentials.

[dawgg]

Tests may still be fair, but only recommending affiliates; http://www.matousec.com/projects/firewall-challenge/results.php#firewalls-ratings
...cheap move

It may be *slightly* more professional if they had "Buy It Now" links rather than "recommendation" for affiliates

It would be even more professional if they didn't have any links or all vendors had links to their website (not affiliate ones)

[Smokey]

Somebody gave me his opinion about the issue:
-{ Quote: "Matousec's integrity, intentions and results are compromised" }-
I can't say it better.

[Tommy]

Admins of well known and seriuos security boards are beginning to ban Matousec reports :thumb:
Wilders should follow this example.

[faenil]

....Online Armor Free in their tests passes Kill 1 ...mine doesn't and I have a newer version...

VERY STRANGE...

[Hugger]

This is either a poorly thought out idea or a poorly written statement on Matousec's part.
Or possibly just poor judgement.
I know that this now makes me a little hesitant to follow his test results/recommendations.
Hugger

[EASTER]

In all honesty i found the toughest scrutiny on security apps originate right here at Wilder's and are taken to task, sometimes heated but with the intent to get right to the core of the truth of matters like this.

kareldjag (sorry if a mispell) is been one of those fierce scrutinizers in his own right and he puts security apps thru some pretty thorough paces himself with results and commentary.

[ggf31416]

I guess most wilders' members live in communist countries.::) :doubt:
Matousec is just trying to get some money from his work.
You can start your own firewall testing site if you don't like it.;)

[Peter2150]

-{ Quote: "I guess most wilders' members live in communist countries.::) :doubt:
Matousec is just trying to get some money from his work.
You can start your own firewall testing site if you don't like it.;)" }-

I don't think it has a thing to do with where we live. When one takes money from the sale of the products you are testing, it certainly calls into question the objectivity of the testing.

[Perman]

Hi,

That wrinkle gives me an excellent idea to embark on a new venture...

to solicit many vendors ( I would get a directory) as possible to offer a guaranteed positive review and an affiliation. ===> scam ? or spam ?

I could become an instant rich man, eh? Not a bad idea !

I would say that Matousec guy has finally shown his true colour--green--.
Money beggar.

US's sub prime loan disaster must have hurt a lot people, including our (ex)honorable friend.

[LoneWolf]

Doesn't change my opinion of Matousec or his leak tests. ::)
Never took his testing or his opinion's very seriously anyways. :thumbd:

[alex_s]

-{ Quote: "I don't think it has a thing to do with where we live. When one takes money from the sale of the products you are testing, it certainly calls into question the objectivity of the testing." }-
TestMyPC does it for free. Still, I'm very hesitant about its objectivity. And comparing these two services I definitely in favour of Matousec. Because .. he uses the same clean system and runs everything with the same criteria. He uses opensource tests. While comletely free testmypc tests every product by _volunteers_ on their PC with unpredictable configuration and unpredictabe criterias.

[alex_s]

-{ Quote: "Doesn't change my opinion of Matousec or his leak tests. ::)
Never took his testing or his opinion's very seriously anyways. :thumbd:" }-
I think it is enough that vendors took it seriously and improved their products. What I see definitely positive. He published BsodHook utility (ShadowHook now), for example. It revealed a lot of bugs in the drivers. Then this was fixed by some vendos -- positive.

There hardly can be something perfect in the World, though.

[Hugger]

Question.
Reading all of our posts I now wonder if there are any reputable firewall testing companies that will give away the results to interested parties? Meaning us.
As an aside, in the past Matousec and I communicated a few times.
I was impressed with his willingness to share his knowledge.
This is an unfortunate decision on his part.
Hugger

[alex_s]

-{ Quote: "Question.
Reading all of our posts I now wonder if there are any reputable firewall testing companies that will give away the results to interested parties? Meaning us.
As an aside, in the past Matousec and I communicated a few times.
I was impressed with his willingness to share his knowledge.
This is an unfortunate decision on his part.
Hugger" }-
I don't dare to judge him. I just wonder, is this really possible that somebody did such hard work for free ? May be somebody crazy who won a lot of money in a lottery and who now dies of boredom ? This is not me, definitely :)

[19monty64]

What's wrong with accepting sponsorship from the companies you're testing??? I'm sure all the AV-comparatives are sponsored by the AV-companies too. I'm sure they get pay-for retesting if they do poorly and a copy of the samples that are going to be used. (sarcasm) ::)

[Einsturzende]

It seems that Matoušec sells product (build) which doesn't exist.
(See Outposts build number on his page and on official one)
Matoušek tested an internal build which is not released yet (and i doubt it will be)...

[Smokey]

-{ Quote: "What's wrong with accepting sponsorship from the companies you're testing??? " }-
Sponsorship can compromise test(results). You can call it a conflict of interests too.

[Pedro]

It's not even the top 2.
And certainly recommending products with a minimum 80% score doesn't seem bad.
If you think these tests are useful or not, it's up to you, but i don't think there's anything wrong here, so far anyway.

[19monty64]

-{ Quote: "Sponsorship can compromise test(results). You can call it a conflict of interests too." }-
That was the reason for the -{ Quote: "(sarcasm) ::)" }-

[19monty64]

-{ Quote: "And certainly recommending products with a minimum 80% score doesn't seem bad." }-
I'd hope to score better than 80% on a test that I had all the Q&A for...

[Pedro]

If i understand you correctly:
Exactly. They don't score 100. The ones that do aren't recommended (no sponsor).

I just think we should not jump to conclusions.

[19monty64]

If I had all the Q&A for a test AND the option for a re-test, anything short of 100% is a BIG disappointment!!! Just imagine a MD about to operate on you, and you find out that he had all the Q&A, paid re-test, and paid the person marking his test, and he scored anything short of 100%.....
But seriously, he's sponsored, so that makes his site a "paid-advertisement" doesn't it???

[Tommy]

Folks, look closer at the test:
1. He is testing applications which are not inteded to be a firewall, like SSM.
2. Matousec does not seam to know about some special features regarding self protection like in Jetico. Even if Jetico is killed (Server and GUI), the rules will still be acomplished or traffic completly blocked. So that does not give a clear result for users who are not aware of those features. Irritating.
3. You can't compare suites from Eset, AVG, etc. with pure firewalls like Oupost, Jetico, etc. Thats like apples and oranges.
4. etc.

So even without the here discussed matter (selling products), which of course makes the results doutable, those test are not professional and correct.

[Atomas31]

-{ Quote: "Folks, look closer at the test:
1. He is testing applications which are no firewalls, like SSM
2. Matousec does not seam to know about some special features regarding self protection like in Jetico. Even if Jetico is killed (Server and GUI), the rules will still be acomplished or traffic completly blocked. So that does not give a clear result for users who are not aware of those features. Irritating.
3. You can't compare suites like NOD with pure firewalls like Oupost, Jetico, etc. Thats like apples and oranges.
4. etc.

So even without the here discussed matter (selling products), which of course makes the results doutable, those test are not professional and correct." }-

I totally agree with you! In fact, it is strange to see Prosecurity wich is not a firewall finishing in third place in a "supposing" firewall testing...

[WilliamP]

If anyone doesn't believe that money will corrupt the outcome,just look at our politicians.

[alex_s]

-{ Quote: "Sponsorship can compromise test(results). You can call it a conflict of interests too." }-
I think it depends. In case the tax is constant, you cannot make every tested product the first. Another point, if you run the test for money you are interested that product failed and was retested again :)

[alex_s]

-{ Quote: "....Online Armor Free in their tests passes Kill 1 ...mine doesn't and I have a newer version...

VERY STRANGE..." }-
Very strange is your resut.

===
Security Software Testing Suite - Kill1
Copyright by Matousec - Transparent security
http://www.matousec.com/

Unable to open process PID 1144 directly, trying handle stealing method ...
Error code: 5
Unable to open process PID 3324 directly, trying handle stealing method ...
Error code: 5

YOUR SYSTEM PASSED THE TEST!
===

Did you configure ssts.conf properly ?

[faenil]

Maybe we're not talking about the same Kill1...isn't it the one in APT 4.0?

[larryb52]

guy doesn't need money, I refuse to look at the list, his results have always IMO been suspect...

[Smokey]

-{ Quote: "Another point, if you run the test for money you are interested that product failed and was retested again :)" }-
Wrong.

To vendors it is obvious that with failed products they will lose existing customers and discourage potential ones.

So if i run a test for money i will be sure the test result will satisfy the paid customer, otherwise the customer will cease his payments to me.

[ErikAlbert]

I follow my own brain and instinct to keep my computer clean. Matousec doesn't tell me anything new, my general readings on the internet told me this already.
Matousec is a waste of time to me, I prefer to read new ideas that might improve my security.

[Perman]

-{ Quote: "Wrong.



So if i run a test for money i will be sure the test result will satisfy the paid customer, " }-
Hi,, Yes, I concur.

I would do trials and errors behind sences, until it is ripe for show and tell.

Any technical writer(this term perhaps is better suited for this ex-new kid on the block--back in summer 0f 2006 ?) who dares to adopt this foolish approach is digging his/here own grave pit.----awaitng the final nail to be spiked onto his c--f--n.

What a moral down turn !

[Smokey]

-{ Quote: "Any technical writer(this term perhaps is better suited for this ex-new kid on the block--back in summer 0f 2006 ?)" }-
As far i know, on May 19, 2006 trooper Matousec announced his existence for the first time on CastleCops, via a spam post:

http://www.castlecops.com/pstp770180-.html#770180

Quote from that post:

-{ Quote: "Our site is called Matousec - Transparent security" }-

-{ Quote: "awaitng the final nail to be spiked onto his c--f--n." }-
Don't worry, he is very occupied with digging his own grave and will succeed with this endeavour:)

[LoneWolf]

-{ Quote: "As far i know, on May 19, 2006 trooper Matousec announced his existence for the first time on CastleCops, via a spam post:

http://www.castlecops.com/pstp770180-.html#770180" }-


Hmmmm.......Snnniiffff.......Yep smells like it was spam.

But like I said in my earlyer post, this does not make me think any less of him or his testing....................Since I only took it with a grain of salt to begin with.

[alex_s]

-{ Quote: "Maybe we're not talking about the same Kill1...isn't it the one in APT 4.0?" }-
Apt 4.2 here. Passed every test twice. 100%. And it was also confirmed by testmypc guy who tested v 119 and failed only kill7 (killtask), which was fixed few weeks ago.

[alex_s]

-{ Quote: "Wrong.

To vendors it is obvious that with failed products they will lose existing customers and discourage potential ones.

So if i run a test for money i will be sure the test result will satisfy the paid customer, otherwise the customer will cease his payments to me." }-

How can you know it is wrong ? We could see that some products were retested very soon after they got the results that didn't satisfy them. So it is likely you are wrong. And what is a sense to satisy customer ? Once a customer is satisfied there is nothing to pay for next time. BTW, in case you suspect that results are incorrect, you can take the tests, the methodology and make a proof results are wrong. Just use HIS methodology, not YOURTH.

[Smokey]

-{ Quote: "How can you know it is wrong ? We could see that some products were retested very soon after they got the results that didn't satisfy them. So it is likely you are wrong. And what is a sense to satisy customer ? Once a customer is satisfied there is nothing to pay for next time. BTW, in case you suspect that results are incorrect, you can take the tests, the methodology and make a proof results are wrong. Just use HIS methodology, not YOURTH." }-
With his commercial move Matousec compromised his credentials.
Everyone is free to put trust in him and his tests, to me he is dispatched and filed as being "dubious case" and therefore a closed chapter.
IMO he and his tests are contaminated territory.

[alex_s]

-{ Quote: "With his commercial move Matousec compromised his credentials.
Everyone is free to put trust in him and his tests, to me he is dispatched and filed as being "dubious case" and therefore a closed chapter.
IMO he and his tests are contaminated territory." }-

I can say only about myself. For me his move is quite natural. He did a lot of very hard and resource/time-consuming work nobody dared or cared to do. He would be crasy not to try to take some compensation. Yes, I know, people do like much when somebody does something for them for free. But I'm not. I know that if somebody does something for free there very likely is some hidden background, and I prefer that cards were open.

[faenil]

-{ Quote: "Apt 4.2 here. Passed every test twice. 100%. And it was also confirmed by testmypc guy who tested v 119 and failed only kill7 (killtask), which was fixed few weeks ago." }-

hey man my online armor is the latest version and fails KILL 7, KILL 10, KILL 1..OA free version 2.1.0.131 ofc...testing in VirtualPC

P.S: Tried with 4.2 as well...

[alex_s]

-{ Quote: "hey man my online armor is the latest version and fails KILL 7, KILL 10, KILL 1..OA free version 2.1.0.131 ofc...testing in VirtualPC

P.S: Tried with 4.2 as well..." }-

This is just impossible. It was checked many times. But if you disabled HIPS, then this is possible.

[faenil]

hey man it IS possible ^^
I can post video prooves ;) Dunno if there's something wrong with my pc...but my OA fails kill 1,7,10...and passes all the rest, so HIPS is not disabled^^

[alex_s]

-{ Quote: "hey man it IS possible ^^
I can post video prooves ;) Dunno if there's something wrong with my pc...but my OA fails kill 1,7,10...and passes all the rest, so HIPS is not disabled^^" }-
Please, do it :) Or better PM with attach, because I'm leaving for 2 weeks and in 2 weeks subject will likely scroll too far :)