New Matousec Firewall Challenge

Exactly. Vendors are trapped, specially those that actually built part of their reputation on leak tests. Every time there is a Matousec revision, if you go to OA and Comodo's forums, there is always the discussion opened by some "disappointed" customer because they lost the 1st place or a "happy" customer, because they grabbed the 1st place from the eternal "enemy".

Then, PC Tools firewall came, with version 4.0.0.45. , which guess what, it went the leaktest way too! And scored fine in Matousec! Which makes Matousec a lucky guy, because the new "Enhanced Security" feature that made PC Tools score so high, was making my PC to freeze completely. And i am not the only case.

They race on who will take the 1st place in Matousec, but in TCP and UDP perf tests, they were loosing ground badly. Of course, there isn't time to fix such secondary matters. What's important is to beat the new leak test... Only God knows how many vulnerabilities the firewalls will have in packet handling, but nobody cares to search because everyone looks at leak tests. Once firewalls were updating to fix "holes" in their filtering. Suddenly today, by miracle, nobody puts in the changelog any firewall fix! They build them perfectly since day1! All you see in changelogs is stuff about the HIPS part.

This is insane. And now we put non firewall products too? Who's next? WinPatrol, Norton Antibot and TF?

If he wants to test behav blockers, then he should do so scientifically. That is, make a new category of "products to challenge" and use specific techniques for them. In this, he should also take into account, the statements of the software houses. For example, Emsisoft here said, that Mamutu will prevent most kill attacks from REAL malware. Now, what if Emsisoft is telling the truth? Doesn't this make Matousec's technique flawed, since it's UNREALISTIC?

Also, it would be nice, to have some "professional" data, in order to be able to assess the risk. He could gather the data for behaviour of say the 2008 in the wild malware and then give what percentage follows the way his leak tests work. Because, you know, if you are going to encounter the "fatal" malware 1 in a 1.000.000, then Mamutu is behaviour blocker 999.999 in a 1.000.000. And NO security application is bulletproof. There is always something that can slip away. In this case, it ceases to be security application? I think not.

Anyway, he should do the test with real malware, since the vendor states that this is how his product works. And put the results, on a NEW category, clean of "firewall" leak tests which Mamutu has no connection to. I mean, since his leak tests or kill tests represent a REAL threat , imitating real malware, he could easily use the REAL malware, delivered in realistic way, right? In this case Emsisoft wouldn't be able to say a word, if Mamutu failed.

 

Excellent post!

 

Someone should test the firewalls against real malware and not leaktests. I would love to see who is the king then.

 

It could only be true in case malware-makers send their examples to Mamutu first, so Mamutu can guarantee it knows most real malware.

 

I agree! Why don't they take the most common malware for example , to give an idea of who is protecting from the most common threat? They can ask antivirus companies for such statistics.

And then, since they claim that their leaktests are imitating the real threats, why don't they use the REAL thing? I mean, if leaktest1 is emulation of trojan1, then where's the problem in using the REAL trojan1, including the ways that it gets delivered? For example, try a drive by from a browser exploit, a mail attachment, clicking on it, etc. This is the weirdest part about leak tests. They claim, that are realistic , because they emulate real threats. So, why don't they use the real threats instead?

Mamutu is supposed to be behav blocker, thus, not based in signatures. So, it's not supposed to need to "know" all the malware before it encounters it.

But hey, all i say is make a "behavioyr blockers challenge" test and then let Matousec prove that! Throw real malware onto all available behav blockers and show the results. I am sure that most people in this forum would be glad for such a test.

I also don't understand why antiviruses are "OK" to have below 40% detection in proactive tests ,but behav blockers must catch everything. Or, are all the antiviruses immune to all termination techniques? Oh, my! What did i just say! If Matousec read this, in the next firewall challenge, he may include Twister, just to show me how awfully "not reccommended" it is! ROFL!

 

I agree, there is some insanity in those tests

But, this doesn't mean there is nothing useful there. The truth, as usual, is somewhere in the middle. My advise to handle Matousec is to ignore rating column, but to look into product report and failed tests description. This info is very useful and can be hardly found in any other sourse.

 

Real malware uses the same tricks. If there is a hole, then it is (or will be) exploited.

Edit: Also I think real malware is not very interesting, because real malware one can collect is known malware, so it is likely against real malware security products should perform better than against unknown techniques. Keep in mind, any test can be easily turned into real malware, just by adding there malicious activity. So if test kills security first it can do then whatever it wishes, for example "format c:".

 

With all the respect, if it is behavior blocker then for me it is bad behavior blocker because it failed to block bad behavior. The only excuse would be if the tests were "whitelisted" and so not blocked. But it's hardly the case.

 

The "truth, as usual lies in the middle" is one of the common sayings that i hate most. If i say you 're a thief, truth doesn't lie in the middle, it lies with you, cause i am slandering you.

Matousec wants to test behav blockers? FINE! He should simply make "Behav blockers challenge", throw them the malware or tests made for them, since he doesn't like using "live" malware and rank them in their own category.

Simple and clean.

Then, here's the opportunity for Matousec to use the real malware and deprive Emsisoft of every argument...

You know, there is one difference in all Matousec's tests, even if we suppose that they are equivalent to malware, and real malware. That the real malware, doesn't always come in a nice exe which you will happily click. The same malware, if delivered in another way, may trip the behaviour blocker. Drive by download, execution wihout user interaction, are different behaviours than clicking it on your own.

You can't simply ignore the manufacturer's instructions of how his product is used and do it the way it suits more yourself.

 

Maybe it is... I don't know. I always prefered TF myself... But you got to give the vendor a chance to prove his statement. And you certainly must rank it amongst its "equals", meaning similar programs. Now he has Ashampoo Firewall (which i use) in the same ranking with Mamutu. What on earth has Ashampoo in common with Mamutu to be in the same ranking?

 

OK, so... real malware is not interesting because it uses "known" techniques. But proof-of-concept leaktests/killtests are extremely interesting and have a huge value since... eh, they use known techniques. Hmmm, makes sense. You know, you can download the test suite.

 

Since some months PC Tools Firewall is going up in Matousec ranking - their annoyances also (see their forum). For them it seems crucial their score in Matousec.

A simple thing - that is a firewall thing - isn't crucial for PC Tools: the option to block all internet traffic. It's far better for their Matousec'score to block executables from other app. even if that is a conflicts source: to them internet traffic isn't a firewall issue because was replaced by Matousec'precepts...

Fuzzfas, congratulations for your post and the good point...

 

This is not the case with Matousec. All the tests are opensourse (unlike CLT, for example, where your only choice is to trust test developer). Since they are opensourse you always can check, recheck and be sure in result. And even if there is a mistake in a code you can point it out.

From his site I have completely different opinion. Matousec looks for the techniques to bypass security software (not only firewalls or behavior blockers) and then publishes his findings. If you don't agree with his policy you can ignore Matousec, his tests and anything else you wish to ignore. I'm not Matousec advocate, so if you ignore him I will hardly be upset. But, I see many useful info on his site for me (as a tech person) and when people bash the whole project just because they don't like "a part", I say they are wrong, because bashing the whole they miss "essential parts".

 

Many techniques become widely known only after the tests were run and published. Level1 tests are really very old and could be addressed many many many times ago.

 

How on earth does this matter when he fails to test relevant firewall features and instead 2/3 of his tests are comprised of this leak test madness and the kill tests plus a bunch of tests that are plain flawed and should be just scrapped altogether since they have nothing to do with security? Publishing the code used for your tests won't make those tests any less junky just because you've published them - those junky tests will produce exact same junky results for everyone, so what? We need relevant and useful results and we don't need vendors focusing on passing some artificial test suite just to pass those.

Except that a product like Mamutu has absolutely no place for "addressing" things like ICMP/TCP/UDP traffic blocking, it also shouldn't scream when whatever app launches IE/Explorer just for the purpose of passing the test. It actually needs to do something malicious to trigger some action - but that's not Matousec's concern in the least, and apparently that's not your concern either. It wouldn't matter for you in the least if the product was vulnerable in tons of ways and failed to do its job properly in tons of ways as long as it'd pass those sacred holy Matousec tests.

 

I dunno. This is political question. I'd like to discuss the technical aspects of the tests. I do not understand what keyloggers do there. I'd also like to have one more column where one could see how many popus you need to pass that or other test. Also I'd like that those popups were reviewed for their informativeness. But these are my dreams, I understand this will hardly be ever done. So what ? So I just use other info, that is available.

PS. The chance for a vendor is s/w behavior. I do not trust in advertisements and other marketing statements. It's this cruel life that made me not to believe vendors' claims. How many times I heard "we are the best, we are the best" in the past ohhh .. no, not now

 

Hey, I already said many times. I think there is SOME INSANITY in his methodology. This is why I DON"T CARE about his methodology. Do not ask me why this or other product was placed there, I DUNNO. But you can ask me why I think some tests are important for any security product

 

I call BS. You've intentionally avoided all the technical aspects and flaws mentioned here and finally resorted to screaming "politics" whenever someone points out specific reasons why Matousec's "firewall" challenge is a broken test.

So, how can you participate in a debate about particular tests when you "don't care about methodology" and "dunno" whether a certain product fits the test or not?

 

I can participate very well. I said that Mamutu failed kill1 and kill2 tests from level1. Kill1 just uses TerminateProcess API. This is the most basic selfdefence test, and as such should be prevented by any security product. Kill2 enumerates the threads in the target process and uses TerminateThread API to kill its threads. This is also very basic termination test that ANY security software MUST block. Do you have any objection to this ?

Edit: If you have any, then this is definitely "know-how", because EVERY decent security product does it
(including behavior blockers like TF and DW).

 

And I said it didn't fail the tests because of this but because the tests include ton of other tests that are plain irrelevant to this category of products. To which your answer has been that you "don't care about the methodology". So, no matter whether it'd pass these two kill tests it'd still fail level 1 tests because the majority of them does not apply to behavior blockers. You can't cherry-pick something and keep your scratched record rhetorics perpetually, that just goes nowhere. The entire inclusion of Mamutu into this test is clearly a brainfart, lay off your "wow but it was killed by two tests" roundabout.

 

I can't dissociate the two.
True, if you look at the test suite, it's nice. But,
1)people are going to see that table.
2)that table depicts apples, oranges and pineapples in an "orange challenge".
3)pineapples are tested for how orange they look.

Look at it this way.
Find a piece of malware that uses that technique. I bet it will be detected by either Prevx/TF/Mamutu, yet they could "fail" that test.
Another way.
Party in your house. A guy takes your ashtray. Your friend HIPS alerts you, test passed. Mamutu doesn't, since it sees the cigarette in his hand: he wants to use the ashtray..

 

OK, let us say, it was killed by a test program, if you like

http://wiki.castlecops.com/HIPS_FAQ

"The most common type of HIPS for home users are behavior blockers. "

So BB is nothing, but a kind of HIPS. BTW, I remember some pure HIPS that performed very well in the tests.

 

Thanks for snipping all the arguments from your "quote" since you clearly are not able to respond them in any way. I'm finished with similar type of debate, don't feel like wasting any more of my time. Either stop this scratched record junk or go away from this thread. Enough noise produced here.

And wine and beer or apples and oranges are the same as well. Of course HIPS performed well in the tests because Matousec is testing HIPS and not firewalls. And the fact that you fail to grok a difference between HIPS and intelligent behaviour blocker will not change anything here. Go download the test and use it against TF or Norton AntiBot, maybe it will enlighten you but I've already lost all my hope.

 

Situation we see is more like:
"you see somebody want's to kill you". You protect yourself by all the means.
"Mamutu doesn't", because it sees police form on the killer.

 

Yes, I do not know what to respond to your "arguments". I don't care about how to call a fail. For me Mamuto failed the two basic tests. All the other is demagogy, sorry.