Matousec 21/02

I don't think the Matousec tests results are completely accurate or relevant. Some of the vendors have modified their firewall simply to get a decent result in this test. For example if you test PC Tools Firewall against other leak tests it does not get nearly as great of a result. Though you should not make your decision of a firewall based on these results, get whichever firewall you like the most based on your needs and preferences, not because of a test. Eg) it shows Comodo as #1 and that's great but not every user wants all the popups that come with it.

 

The only thing I do not understand,from years of lurking here,prior to joining,is why this test allways gets so many peoples blood up?

The tests are what they are.

I doubt many people switch Firewalls based on them,and it always gets morphed into arguments about DefenceWall,Antivirus Etc.

My guy,Kerio 2.1.5 is NEVER going to show high,and I have no intention of switching to the "Latest Greatest Per Matousec" simply because something else
is light years better, based on a set of criteria that does not concern me.
My choice does exactly what I want it to do. (need is subjective)

Still,I always read the results,and bet on which part will cause the most acrimony.

"Zeros and "Ones",Brothers,just "zeros" and "ones".

 

Is this a good test to refer to?

 

Only if you understand how the ranking works and you ONLY are looking for how perform the HIPS component of the security software analyzed

 

It is not good it is not bad, it is simply irrelevant. Effectiveness of HIPS should be tested based on real threats (worms, trojan, bot, etc) like it is done for antivirus and antispyware programs.

Matousec compiled tests only measure how good or how bad a product score against...... matousec compiled tests.

 

Also a test to show how easy a security software can be bypass by a hacker without using self-programmed tools?

 

IMO it's still a good test as it shows some HIPS products can still be bypassed.

 

Hear, hear.
Very nicely put, fax.

LOL

 

That's right in principle but the problem with testing HIPS either against real threats is that the protection they offer is largely subjective.

For example if malware 'A' triggers a pop-up that shows an anomaly has been detected,yet the warning would only be clear to an expert user,then does this count as a pass or a fail? For me it's far more difficult to get any sort of definitive answer as to the efficacy of a HIPS as opposed to an AV test where a simple detect/non detect can be judged.It could become a logistical nightmare if you ended up with some sort of sliding scale of pop-ups to user skill level ratio.

 

Yes, correct but the same will apply to the matousec tests if run by a clueless user, i.e. he/she will allow all and fail them.

 

Quite so.
I'm not saying that the Matousec tests are particularly realistic,just that it's very difficult to test HIPS in a way that accurately reflects the real world,since much of the protection is provided by the user themselves.

 

Depends on how you test.
I did such a test last year with real threats, all programs with out of the box settings and denied all prompts after execution of the malware.
So there was not much subjectivity, as all programs had the same chance to succeed.

However, some Matousec heroes failed terribly, just like they would fail terribly if Matousec would test with out of the box settings. The ranking would be totally different.

Apart from other flaws - this testing only with highest setting makes Matousec's results very misleading for many, as they most likely assume - this is the protection I will get right after installation if I use this app.

For the very involved vendors it's just a promotional event, sometimes a very cynical one.
I mean adapting a security product to synthetic tests is strange enough by itself, but when you see that this nonsense gets highest priority, even higher than protecting against real threats... what a freak parade.

Cheers

 

Well said, subset.

 

Your tests illustrated a couple of things at least.That often the 'out of the box' settings on these products are more about reducing noise than maximum protection and the most important factor with a lot of them is the user themselves since expert users will tweak them to much higher levels.

You're quite correct in saying that by Matousec testing on the highest settings is misleading,since many come pre-configured with far less than full protection.As for the vendors I can see why showing themselves in a high position on these tests is a good marketing strategy,however it's achieved.

 

Very good point and very true. And I think many users would install and forget, leaving things in the out of the box state.

 

All of the tests I see are run on XP. I would like to see some testing with a more up to date OS. I haven't used XP in 3 years.

 

I think that they are working now in win7x64 test

 

Has there ever been such a test that was "relevant?" I suppose if a firewall does not block some threat that is not relevant. If it does so the same.

I wish those who really do know so much would conduct such a test so that we could have some "relevant" information on which to base a decision. I guess that will never happen, but all tests will be seriously flawed in the minds of some. Yet they do not do tests, and it is easier to be a critic than a doer.

Regards,
Jerry

 

Lol, you guys should relax everyone have their own opinions

 

Amen

 

If this test is useless where's a good firewall testing site with results? As far as I'm aware this is about as close as it gets lol.

 

There are none. Its very time consuming and resource intensive to test properly firewalls including their packet filtering performances