Matousec Test Updated

Ya, they are not logic argoments, but only dialectic answers to elude the focus of the post.

 

LMAO. Sorry to help drive this thread further off topic, but that is the best comeback I've seen in a long time. Elegant. Polite. Awesome.

 

I'm also with Mike on this one.

The other thing to remember about Matousec is that he now has a commercial owner. By doing what he does in the way that he does keeps people in places like this talking about it. From a marketing perspective this can only be judged as successful. It's just like the TV soap powder commercial that everyone hates but everyone talks about - mission accomplished!

In truth nothing on Matousec will protect you unless you match the best choice with your individual requirements and ability level, configure it correctly and adopt other practices to minimize risk.

It's also pleasing to see the CEO of a major vendor "mixing it" with the guys here and offering constructive input unlike some others who issue a tirade of semi abuse every few months and then expect all their users with blue screens to fade into the sunset.

 

I guess that now it makes sense to test such products. Now the challenge is called "Proactive Security Challenge". So, not only dedicated to firewall testing, even though in the past firewalls + hips were tested, against those who are pure firewalls.

But, I believe that results shouldn't be mixed.

For example:

Online Armor Personal Firewall 3.5.0.14 is 1st. But, it includes firewall + HIPS.

ESET Smart Security 4.0.417.0 is 24th. As far as I know, it doesn't include a HIPS.

But, looking at those results, does it mean that ESET Smart Security has a firewall necessarily worse than Online Armor's? (I'm not saying one is better than the other, and vice-versa. Only that what can a random visitor think of such results?)

Truth is, as a firewall, Eset's could be much better than Online Armor. The only problem is that it has no HIPS. So, it won't, from a start point, even reach a near result.

I believe Matousec should make tests for:

Firewalls
Firewalls + HIPS
HIPS
Behavior Blockers

Not mixed all.

 

My opinion is matou tests are OK, I'd say they are the best of a kind. You can hardly come with the better examples. But some people, who demand much, not doing anything useful for the community, always spoil a fun, by telling how "this or that should be done" and what "these and those should do".

 

Nope. It only means ESET Smart Security doesn't pass the given set of the tests better than OA. Theoretically it can pass better some other set of the tests in some other project, which would mean that it passes better some other set of the tests.

Nowhere on matousec site I noticed he claims it to be absolute score. It only reflects what matousec personally regards as important concerning proactive security and this is clearly pointed out if one cares to read.

Also, what you can learn about ESET from the tests is:

Breakout2 0 % FAILED
Coat 100 % PASSED
ECHOtest 100 % PASSED

Kill1 0 % FAILED
^^^^^^^^^^
Kill2 0 % FAILED
^^^^^^^^^^

Leaktest 100 % PASSED
Tooleaky 0 % FAILED
Wallbreaker1 0 % FAILED
Yalta 0 % FAILED

Even not caring much of the leaks you can see ESET has weak selfdefence. This does mean it's not too hard to prepare exploit to attack a computer protected by ESET.

 

I agree, absolutely. But, my point was, say, I'm a complete newbie working with computers and search the web to see what others have to say about XYZ security product. I come across those results, and lets imagine that, as a newbie, I'm looking for some table results telling which one's the best. I see that Online Armor is the winner over there, and Eset's firewall is at the bottom.

And, at this precise moment, I have no idea what HIPS are. I'm just looking for a firewall, which stands its ground. According to Matousec's test, Eset's doesn't. And, I'm only mentioning the table, and not that somewhere in the site it is mentioned.

I see that Online Armor is the best and get it. Then, I see some odd questions being done to me, etc, and I don't even know what it is. All I ever wanted was a firewall.

This is just my personal opinion, but those tests should be done in sets, as in, pure firewalls against each other, firewalls + HIPS against each other, etc. Obviously, mentioning that XYZ is a firewall or a firewall + HIPS, etc.

 

Yes, this should be taken into consideration.
I also like the fact that he took a definitive stand against the inclusion of Ask Toolbar, just like BillP, the vendor of the very good WinPatrol.
It is nice to see some vendors are willing to draw a line in the sand which they will not cross, unlike some others.

I am thinking I was too hasty in excluding OA from the list of software to evaluate ; I need to read some more posts to determine whether I should examine OA free first or just go for the Pro.

 

I agree with this.
I am left wondering if this and the tactic of mixing old and new test results is done on purpose so as to "persuade" vendors to pay frequently for updated results.

 

I think you want packet filter, personal FW must distinguish which app. made network request, if you try esets "FW" you will see some application that made request is intercepted properly and some of them can actually bypass esets FW (leaktests and some malware), so estets FW is maybe perfect packet filter but very bad application firewall...

every application should be retested within 3 months free of charge, with exception of appz which do not change their build number from last test, also application can be retested by vendor request, in that case vendor will pay some amount of money

 

You're right about that, but I was talking merely about filtering inbound traffic.

That makes me think why Windows own firewall isn't tested. Yes, it would fail protecting against outbound traffic, but it would show is as great as any other to filter inbound traffic.

I guess such testers aren't really interesting testing it. If people start realizing Windows own firewall does the job, quite nicely, filtering inbound traffic, then I guess some would start loosing money.

I'm aware that most firewalls are evolving to something more than just a firewall, by including HIPS, and that's why I mentioned, as a personal view, that the tests should be done separately.

Is more than obvious that a firewall designed to filter inbound traffic, will stay way behind those other "firewalls". So, is an unfair testing, IMHO.

But, again, allow me to reiterate is just how I see it.


Regards

 

I notice that every product has failed the Level 9 Crash 7 test.

Are any of these tests available for public to test on their pc?

 

I think this may be what you are looking for

http://www.matousec.com/projects/security-software-testing-suite/

 

We're lucky, OA is the best, otherwise these results could be misleading
There is no doubt about it, Matousec is a lot about marketing for vendors. Overall the methodology he uses is good (open, source, reproducible, tested to same standard) but some things that he does are just wrong. The scorecard based on the different sets of tests I don't like, and he'd be better to do all tests on all products, rather than cut them off at a level. It's theoretically possible that product X could fail some level 2 tests, but then pass all level 3-->10. But, it's a lot of work to do these tests, so I can see why it's done the way it is done.

We nearly did it. It would have been the worst mistake ever - but we were very, very close to doing it. The money involved is just huge. "everyone else" is doing it - justify it how I like, but really - it would have made us a fortune. When I wrote on my blog it could have paid for a (used) Ferrari - I wasn't joking. Unfortunately, the price to be paid for those riches would have been a lot of our users hating us. That's never going to be good.

I'd have a look at the free version first and see if you like it. OA works a bit differently to many products you may be familiar with. If you think that it fits your style, take a play with the Pro version, or you could wait till next week and have a play with the new ++ version which includes the EMSI/Ikarus engine.

They are all available on the Matousec site. The latest beta of OA passes Crash7. This test is not -really- so much about secuirty, it just gobbles up memory - but since controlling runaway memory would contribute to system stability, we decided to include it.

 

I tried the Crash 7 test. I can't even get crash7.exe to execute and run even with md disabled. Strange.

 

Most of the tests crash on Vista due to some specific code that disables usermode hooks (if any). They are designed to work on XP.

===
Methodology and rules
Installation and configuration

The tested products are installed on Windows XP Service Pack 3 with Internet Explorer 8 set as the default browser.
===

 

Some of you don't get it. Matousec makes it clear the criteria he requires in personal firewalls.

It is all explained here:

Design of ideal personal firewall

Some excerpts:

Self-protection
This is rule no. 1 for all security products, not only for personal firewalls. No matter the perfection of other features, if the firewall is not able to secure itself it is useless.

Process protection
Every privileged process must be protected against several dangerous actions. Firstly, no malicious application can terminate the process. Secondly, it must not be possible to modify its code or data. Thirdly, it must not be possible to execute any code in a context of any privileged process. This point also includes DLL injection.

File and component protection
The protection of files is very close to Process protection. If a malicious code is able to replace files of privileged applications it is equivalent to modify their code flow when they run.

Service protection
Since a part of the firewall is usually implemented as a system service the protection of system services is also necessary. But it is not only the firewall component that has to be protected. To install a new service is easy way for malware how to persist in the system because system services can be set to run every system start. What is more, a malicious service can be dangerous also because it runs even if no user is logged on.

Protection of other system resources
There are also different system resources and objects in Windows operating systems. Some of them can be dangerous if they are controlled by malware. One of these objects is a well known section '\Device\PhysicalMemory' which can be used to gain the complete control of the system if it is not protected.

Parent process control
Some privileged processes can be misused to execute privilege action if they are run with specific command line arguments. Many firewalls do not distinguish between the execution of privileged and unprivileged processes.

No ring3 hooks
Ring3 hooks must not be used to restrict behaviour of unknown applications. They can be used very rarely to modify or control the behaviour of privileged applications that are guaranteed not to bypass ring3 hooks.

This is why pure packet filtering firewalls are not separated from those that include HIPS features.

The scores are based on all the criteria he expects to see in personal firewalls. This is why there is no categorization of these products. Right or wrong, he expects all personal firewalls to include a lot more than just packet filtering capabilities.

 

As for pure firewall,those tests shouldn't be used.Those tests should be used to test HIPS like COMODO and so on.As for Lns and other pure firewalls,it is unfair.Some people dislike or are disable to use HIPS,they only like to use pure firewall.

 

Hm. But what is unfair ? If people like to use pure firewall let them use pure firewall. Do they really need matousec approval ?

 

What is unfair, is that, when looking at the table's results, you see results for "Bananas", "Apples", and "Oranges".

Now, a user going to Matousec's site and sees the results, but there's no mention that the results over there are for those different "fruits".

So, as a casual computer user, looking some firewall (All I want is a firewall, which is something that filters inbound traffic. That's all I ever heard about what a firewall is.), I see that one of those "Oranges" have a really low score. Now, does it perform a really bad job protecting your health? And, I also see that those "Bananas" also have a lower score. Does it also mean they don't do that of a great job protecting your health? Do they lack vitamins?
But, I do see that those "Apples" are on top. Now, they really must be the best protecting my health.

What I don't actually know is that, those "Apples", are actually a combination of "Bananas + Oranges". All I want is to check results for "Bananas". Worse, those "Oranges" aren't even firewalls, they're what people call HIPS.

So, if someone wants to test the different levels of protection a specific "fruit" gives to our bodies, then compare "Bananas" against "Bananas", "Bananas + Oranges" against "Bananas + Oranges" and "Oranges" against "Oranges", and report the results in different tables, mentioning what each if for.

Initially those tests are simply called as a Firewall Challenge or something like that. This was deceiving. We all know, here, that, for example, Mamutu is no firewall, so it won't filter inbound traffic. We all know that some of those firewalls are pure firewalls (filtering inbound traffic), so they are no match to those other "firewalls".

Now, the test was renamed to Proactive Security test. At least, is not that deceiving, still it should report the results in separate tables.

 

If the public was not interested/hyped up in these tests, and threads like this help, I wonder if the guy would be able to persuade the software makers to pay up for the privilege and or the bugs he finds

 

But yes. Who does the job commands the rules. Isn't it fair ?

 

A cynical view but given that most users pay no heed to version numbers and just look at the product name when appraising these tests it's all too believable.

 

Yes, the one who does the job, rules. Fair? Not really. Is not fair to rate Mamutu 1.7.0.23 2% and say "Not recommended". Everything is in the same basket.

Matousec is making it seem a really bad product. The same for ThreatFire.

Would you test cars, motorcycles and trucks, regarding their efficiency (in all fronts), and put them all in the same basket, and say: "Hey, don't get this product, because, well, it sucks. Why? Well, for starters, it won't allow you to load great amounts of charge". Really? Well, it's a motorcycle.
Or "Don't get this, as it is slower, compared to others." Really? Well, it is a truck.

So testing Firewalls, Firewalls + HIPS and Behavior Blockers, and then put them all in the same basket, and say "Not recommended", really isn't fair.

 

Yes,I totally agree with m00nbl00d.Many and many people want to know which firewall is the best,I am usually asked about this question.They like to see the firewall test.And matousec firewall test is very famous.Many people think if matousec said the firewall bad,they will think the firewall can't become a Network Defender.They will give up that firewall.But they don't know the matousec tests a HIPS not a pure firewall.That is the reason why I said it is unfair.