New Matousec Firewall Challenge

You are judging Mamutu's strength, i'm trying to explain that Mamutu's class works differently with an example.
Testing if Mamutu detects the guy grabbing some object of yours is not good.

 

To quote myself:

The reason why TF or AntiBot are not there is that Symantec would sue the sh*t out of Matousec's ass for including them in such "firewall" test and labeling them as totally failing products.

Bye.

 

But I have never said that Mumutu was tested in the correct testing

I just said that being tested in a wrong testing it showed VERY WEAK selfdefence, which for me is enough to make some conclusions. Mamutu is BB, which is actually kind of HIPS. Let us say all the BB I know can protect themselves from termination. And I'm sure in some short time Mamutu will do the same.

 

You basically say, since nothing is infallible and Mamutu can miss malware, that malware can easily terminate Mamutu.
I agree, UNLESS Mamutu actually allowed the termination since it detects that it's actually you clicking a button to terminate it. Only Emsisoft can clarify that definitely, and it seems he already did that:

 

Any vendor can do the same. He can say "we only failed the test because it was a test, not a real malware". But if you take this statement you should forget about any testing forever and trust those magazines that run likely "against real malware" tests, but actually nobody knows what do they really do. We should understand that no testing can be complete and absolutely objective. Still I welcome efforts people put in this field. Beside of many "insanity" we have a lot of useful info as a result. Then it's a task for thoughtful mind to separate "rabbish" from "useful info".

 

Wrong. Actually there are respected antivirus tests that make the malware samples available to the vendors to be able to verify them. Whether they reflect the real world's reality or not can be disputed, but at least they don't ask antivirus vendors to filter TCP/UDP traffic and similar junk.

 

No they can't.

 

Do you think you can stop them from saying ?

They can, and they did.

BTW, about real malware. Any good malware database contains over 1 000 000 of examples. Do you think this is possible to test a set of products aganst even 1/10 part of known real malware ? Then what criteria to use to select testing set ? And finally, do you think anybody will do this crazy work for free ?

 

Please, stop trolling. No, you really can't say as a FW vendor that you don't care about blocking ICMP/TCP/UDP traffic without looking like a complete idiot. OTOH you are perfectly entitled to say this if you sell a behaviour blocker application and if the tester checks your product for such feature then he'll in fact look like an idiot himself. Unfortunately, there are some users who will praise the tester for his infinite wisdom regardless, those are best ignored both by the vendors and general audience.

 

Nope.
Ok, we found a dead end. Let me try another route.

If you agree Mamutu wasn't built to alert you on a single action, why do you doubt Emsisoft?

 

I think a lot of this type of legitimate doubts and suspicions could easily be resolved simply by, if he (they=others) would document their tests openly to the public via a url to a Video File review of those tests for us to see. (provided they don't doctor the video to display something which is not from something which really is) (tongue in cheek )

 

Just noticed this part.

But what does that have to do with Matousec's tests?
Do you think they should build programs made to be tested, or to work?

 

Because:
1.) I do not trust anybody, I trust just experience and reproducable tests (including reproducable tests with real malware I have VM for).
2.) This single alert we talk about is crucial. If this alert is not shown we have no chance to see other alerts (because there is nothing to show them).
3.) Any decent security MUST protect itself from accidental termination. You only should be able to shutdown security program by clicking "shutdown" and then "yes, I'm sure I want to shutdown"

 

I do not see any contradiction. Program can work well and pass the tests well. I have no idea why some people do not see those things are associated. Most of the tests were built on real malware techniques. Just a few are questionable (I'd say artificial). Some new tests introduce REAL ways to bypass security. Let us take just an example:
Method description:

* Open a helper process.
* Allocate memory in the helper process and write the infection code into it.
* Use CreateRemoteThread function to create a remote thread that executes the infection code
inside the helper process.
* The infection code tries to open and terminate the target processes.
* The whole procedure is executed with helper processes "services.exe", "winlogon.exe", "lsass.exe" and "csrss.exe".
* This test works with a list of processes and reports success if in at least one process was terminated.

This is pretty real technique to bypass protection (here it just terminates, but it can do anything).

 

Anyway, Mamutu's inclusion in this "challenge" is wrong.

 

Well, Matousec FINALLY fixed there mistake for Comodo and apologized. All I can say is, I agree Matousec should use REAL MALWARE to test these Firewalls & HIPS, these tests provided by Matousec have no meaning in real life, Geez... And they HAVE TO change the testing methodology. Look how many flaws are in the current testing methodology:
http://forums.comodo.com/leak_testi...ew_matousec_firewall_challenge-t30896.45.html (Reply #59).

Please David/Matousec. Change you're methodology and change it fast. my opinion... Now I will let you all argue about it! Cya!

 

Then he should change the name from "Firewall challenge" to "Security products vs Matousec's challenge"...

In fact i DO ignore them, or i wouldn't be using the "not reccommended" Ashampoo firewall right now. But this doesn't mean that i must shut up if i see something as unfair. I would say the same if AV comparatives made on demand test of antivirus and threw also behav blockers in it. It's like ranking together street cars with 4x4. They aren't the same thing, just because they are "cars".

He may have useful info. But his methodology leaves much to be desired and his final rankings, after this one, are completely non scientific.

I can make a site of my own and call it "AV challenge". Then throw in HIPS, firewalls, behav blockers, Sanboxies and Antivirus , and come up with a final ranking, putting them all in the same list. Would that be scientific? Or would everyone say "Oh, Fuzzfas, poor amateur, what is he smoking?".

That's why in AV comparatives they only test... AVs... Because , in the final ranking, they can put products that are supposed to do the same job in the same way, and so, can be put in the same rank list.

Anyway, this discussion is enough for me. After all, i am not an Emsisoft user. I am just tired of seeing newbies in various fora having Matousec as gospel. After a week they ll start "noooo! Don't install Mamutu, it's trash! Look at the proof at Matousec!".


- Make a firewall test. Put all firewalls inside and test them with tests made for firewalls.

- Make an AV test. Put only AVs in and test them in the same way against malware.

- Make behav blockers test. Put only behav blockers and test them against tests that are made for behav blockers and not for classical hips or avs or firewalls. Or even better, against malware, since they claim that they trip on real malicious behaviour.

Get all 3 results and rank them in 3 different lists. That is the minimum way to do it , if you want to be called anything near a professionist.

And if you want to make an even more scientific job, ask for major av vendors their statistics about malware frequency for 2008 and test with real malware. And then you can give the customer the best way to decide which product is more likely to protect him. For example:

You test Mamutu, TF and Antibot with REAL malware, so that the vendors won't protest.

Then you put :

Malware No1 (High frequency according to Norton Labs): Mamutu pass, TF pass, Antibot pass.

Malware No2 (medium frequency): Mamutu fail, TF pass, AB pass.

Malware No3 (low frequency)...

And so on, covering malware with different infecting technique and different frequency. At the end one can decide what he wants more. For example, between one behav blocker that fails a very common malware but passes a very rare one and one that does the opposite, the customer would have an idea, of what HE thinks is the best for him.

But that would be too scientific job to ask for...

P.S. : If Emsisoft was a big company not worrying about spending money, i think, that now Matousec would face legal action and i also think that he would loose the trial and would be paying a lot of $$$. Maybe if he includes Norton Antibot (which would only be fair at this point!), Symantec will take care of him.

 

I propose for the next Matousec test to include all major AVs, and test them together with the firewalls. After all, many AVs do flag some leaktests as threats, so some would actually score well!!!

And of course put them all in the final ranking alltogether! Please Matousec, do that! Some AVs may be killed after all! I am sure all major AV companies won't mind seeing their products in the "red zone"! After all, if some AVs get killed by his tests, this would suffise to legitimize such a test! Or maybe not? Well, the judge will tell!

 

If you think he "should", you'd better say it to him, not to me, please

 

1) I am pretty sure he reads this forum. You know, he used to be a member here... (maybe he still is! who knows!)
2) I can say it in this forum too. Otherwise we should close the thread alltogether and talk about it only through mails directly to Matousec. If you think i can't or that i shouldn't, please complain about it to the moderators.
3) Maybe we should close all threads were people make suggestions, wishes,features, complaints for various programs and direct them directly to the software houses. Or not?


Regards.

 

Of course you can ! There is just not much sense to say it to me. I can't help it.

It would sound better like: "Hey, Matou ! If you are reading this, <censored> <censored> and <cernsored>, then you SHOULD do <censored> <censored> and <censored> !!!

 

It is a harmful action but if you check out the root of those it is really no wonder, imho. There is a huge difference between western and eastern regions the tone is so different.

maybe, maybe not..

This is a very old idea that they probably have stolen from me but maybe there is a slight likeliness that they only had the same idea but I had the idea much much early then them approx 8 years ago, hahaha.

 

A real test would be to embed the malware in a browser or plug-in exploit, put it on a test site, and let people test blocking the malware from getting onto the computer in the first place.

----
rich

 

In any case, this kind of testing is just fine for applications like Comodo IS. That's not the problem me thinks.

 

4 - Make an HIPS test


Matousec should be do it for all the future tests, distinguishing the softwares to test for kind and developing different and focalized tests . It's a problem of testing procedures and of clarity and mean of the results.