New Matousec Firewall Challenge

I'm not going to even bother replying to alex_s comment's to be honest, I dunno what "world" he is on. We will wait till the new CIS version next Tuesday.

 

Actually he should do a lot more, for example to retest everything with all the 84 tests. But it takes a lot of time, you know. And since this is his private time he can spend it as he wishes, not as you or me wish. BTW, as far as I know some time ago Melih declared they do not care about leaktests anymore. So what this buzz is for ?

Another thing I remember, Comodo was very angry that some companies fixed their bugs and submitted paid retest. Then the rules were changed that every single product can not be tested faster than in 6 monthes. So I guess there is no way for Comodo to be retested.

 

Can you provide any proof of this

 

http://forums.comodo.com/leak_testingattacksvulnerability_research-b55.0/
http://www.matousec.com

I'm not sure which statement you need poof for, but all you need you can find there.

Though, now it is 2 times in 6 monthes. My bad. But in any case it cannot be done right now.

 

Double post.

 

I remember Melihs posts about this. He was annoyed with Matousecs business model of pitting one against another and taking payments to reveal failings and have a retest thereby moving their product up the list.

Basically if a company refuses to pay him, they slide down his league table.

Just how many of his tests are relevant to the average home PC user anyway?

 

How many times average driver gets use of air-bugs ? Can it be said they are irrelevant ?

This is security. Malware is not just a fixed set of files and exploits, it evolves all the time. If there is a "weak" point in security it CAN be exploited in any unpredictable moment.

 

Air bags can be very usefull if you lose control like in a chain reaction type accident as well as others types of accidents.

That is why AV has heurisics.

 

Stop making illogical connections. Leak tests cannot be compared to air "bugs" [sic]

 

I myself have only commented on the methods... Lets see, you have a site with a "wine challenge" project. Initially, it's been a pretty cool site, wildly respected and gave out useful information about the quality of various wines to consumers.

Eventually, you get bored by the boring testing (the color, aroma and taste of the wine doesn't change so much during the time) - so you alter the tests so most of the focus goes on whether the wine bottle can be broken and whether the cork leaks... People get confused, because they don't get much information on the wine quality itself, but instead there's a whole lot of tests about the quality of packaging. But you say - what the heck, if the bottle can be broken or leaks, the consumer won't get any wine, stop complaining about my tests just because you use crappy bottles and cork.

Eventually, you get bored even more because there are limited number of wine brands available, so you throw a bunch of beers into the test - after all, both wine a beer contains alcohol. Then you go and give out a "shitty" rating to the beers, because wow, what a junk, it doesn't taste like wine at all, it doesn't come with a cork and some of them even don't come in bottles but in cans instead... Wow, what a junky wine.

So, in the end the consumer gets a totally irrelevant chart which instead of the quality of wine compares the quality of packaging of random alcoholic drinks. The beer vendors get furious, the wine consumers get pissed off and eventually noone visits your site and people make sarcastic jokes about it instead.

And, that was the end of Matousec "firewall" challenge. Good night, kids.

 

+++++!!!

 

First of all i want to say that thanks to matousec all win.
Thanks to him comodo has discover today some bugs and they are going to release an update the next week to get 100% score.

Other apps like like outpost firewall, OA, ZoneAlarm,... take this test for improve their software, thanks to this the final user wins.

For all of this, I want to give thanks to Matousec for his work.

There is a poll in his web about firewalls, hips... and he for testing always choose the products in the firts places, mamutu was an exception because some user send an email to him.

It's normal that he want to earn money with this, "nobody" works for free and his test are the most profesional that we can find on internet, if somebody can do better i hope that he do.

Please stop to criticize and send an email to him with your opinions.

I want to ask.
What kind of security offers Mamutu if it can not be able to block that another application close it, like a virus or any malware.?

This leaktest represents "all" the ways that the virus, malware... take to infect a computer. No?
What kind of behaviour blocker is Mamutu if it can be able to detects this ways...?

Maybe this test are not for mamutu but I think that emisoft have a lot of to learn from this results and at least fix bug (because this in a security software is a bug) that another app can close it.

 

Well, I beg to differ... In the end, it's the end user that suffers.

- The nontechnical ones get a false impression from comparing wine to beer (to continue in the above analogy; the vendors suffer from this as well, explained below.)

- The technically savvy users suffer because there's basically not a single test left that'd test the original firewall functionality - do the SPI features work as they should? Noone can tell from those termination/leaktest results. Do ARP cache protection features work as they should? Noone can tell. If you opt to do some filtering on a MAC address basis, do those features work as they should? Matousec won't tell you. Does the firewall affect network performance and to what degree? Even this test has been dropped.

- Both technically savvy and ordinary users suffer since some firewall vendors focus on passing the tests instead of fixing and extending the functionality, since that was the users demand. They take Matousec as their mantra. If it sucks in Matousec's test, the product must be junk. To continue the wine analogy yet further, in the end you find out that there's no longer a good wine available, instead there's plenty of uniform wines in leakproof unbreakable bottles.

It's not normal to ask for money to fix your faulty tests.

Erm, no... The leaktests technically require malware to be already present on your system, meaning another layer of your defense (such as AV) already failed and the computer is already infected. Then there are completely broken tests already mentioned before in this thread that have nothing to do with real security (like system shutdown, socket sniffing or trying to run the system out of memory).

 

As I have described to Matousec several times, Mamutu is made to block real malware samples, not to pass leak and performance tests.

The product purpose is different to firewalls and HIPS.

Firewalls and HIPS are made to alert every single suspicious action without combining them to alert a bad behavior.

Behavior Blockers like Mamutu are made to show the least possible number of alerts. Mamutu alerts programs that are most likely real malware. We're working hard to NOT alert good programs. E.g. if a program runs visibly (like a test tool) the malware scoring is much lower than if it runs hidden (like real malware). Mamutu does not block shutting down the program by the user (a test tool is more or less the same, manual action), but Mamutu blocks shutting down by real malware.

That's why I told matousec to test with real malware. Mamutu is proven to be one of the best behavior blockers beside Threatfire, Antibot and others (which are both missing in the 'firewall' test btw.) to block real in the wild malware.

While all security programs can be improved when it comes to self protection e.g., most real malware samples are cought long before they can even try to kill Mamutu.

Matousec's test does not help us a lot. It does not help us when they tell us that Mamutu does not block TCP, UDP and ICMP traffic. We already know. And we don't have plans to add such features as the program is NOT a firewall.

 

Thanks for explain me the diference, them I supose that all the antivirus has a behavior blockers. I am waiting to the 64bit version of mamutu for try it.

 

Excellent posts by DoctorNotor and Emsisoft.

In fact, once the firewalls were... firewalls. And people would search for vulnerabilities in the firewalling process. Since Matousec started the "challenge", firewalls, became HIPS with a bit of a firewall... PerTCP and UDP tests of most of the products sucked, but hey, they pass leak tests and that's more important. In fact perf UDP and TCP tests were dropped.

So, once, there were HIPS products doing the HIPS and firewalls doing the firewall. Now, because vendors want to take the "Matousec prize", firewalls became HIPS too. And if this goes on, behaviour blockers will become classical HIPS too.

The utility of leak tests has been argued many times in this forum. One point very important made by Emsisoft, apart the fact that firewalls shouldn't be Hips in the first place, is that you launch leak tests on your own and aren't real malware.

Another thing that should be interesting to know, would be, what is the actual percentage of malware using each of the methods used in leak tests. Because leak tests are POC. It doesn't mean that all these techniques are widespread in the wild. Because theoretically, a car's windshield may brake if hit by meteorite. Yet, nobody cares that his windshield can't resist such an impact, because it is highly improbable that this will happen.

The funny thing is that in order to stop malware that you didn't execute yourself, all you need is simply execution protection (like the good old PG FREE). If you do try to install something that contains malware, most probably you will have put the hips in "installation mode" and it won't see anything. And oddly enough, a behav blocker in this case, is your best bet.

Basically, if a user only installes programs from reputable sources, all the leak test thing is futile. And simple execution protection can protect from drive by malware execution.

 

I am very hard pressed to even give mediocre credit to other Behavioral Blockers like TF and definitely never AntiBot which failed me miserably enough to learn a wild cat never sheds his spots, so is Symantec in my book from experiences that continue to fail the grade consistently in more ways then one.

TF is IMO just simply lazy right now, but it's programming has great promise for the future should they decide to exercise some of that interest.

And i agree, i would love to see the other BB's run thru the same tests. MAMUTU continues to have a great future in spite of this particular result offered from matousec.

And the jury is still out for TF afaik. Symantec Norton™ AntiBot is out of their league again AFAIK and should just continue to stick to their other bloated materials.

 

I asked Matousec's team why they didn't test OA free. This is their response:

 

and I think too many people on here are paranoid and scare themselves into thinking every time they go online they're in real danger, filling their pants in the meantime. Just what great national secrets are you keeping from the world there on your PC?

I used windows firewall for ages (years) with no router, no problems.

I now use comodo and a router, still no problems. If I want to visit the darker side of the net I'll do it in sandboxie. I don't use cracks or keygens so where are these problem Matousecs tests allegedly show, coming from?

We should have a poll and see just how many people here actually suffer from each and every one of his 'problems'.

In fact it would be informative if he quoted real world figures of the amount of infections/problems each of his tests causes out there in internetland.

To me, the scary headlines cause more problems to more people than the actual security issues he exposes.

 

It's a good idea. I 'd be curious to see how many members have actually been "saved" by classical hips, AFTER the initial stage of allowing the initial exe to execute. Because that's something PG Free could do too. So, it would be interesting to see, how many members have been saved once you allow the malware to execute, in cases where you don't know that you are actually executing malware.

Nah, that would be too scientific to divulge... I would be glad with a simple percentage of the "in the wild" malware entered in the library of a major antivirus brand in 2008 , to see what percentage uses each of the leak tests behaviour.

Of course. But scary headlines also bring more people to spend money for protection they don't really need. (see 10 different programs).

 

Very good, except you forgot one thing: user looks for wine, goes away knowing which beer to buy.

 

I may just have to remove Look N Stop from my computer to avoid a conflict between my Mamutu-Look N Stop firewall configurations.. ..just kidding...

 

Wow, not only you run 2 firewalls , but also 2 of the crappiest ones! Both are in the dead zone, err... i mean in the red area.

You are not safe! You 're in double peril! Quickly! Uninstall! Uninstall!

 

Gees!!, thanks for the heads up. I'm in the dreaded Matousec RED zone... .

 

My response from Matousec Support as well about Mamutu
being removed from their list