Did you choose your firewall according to Matousec ?

I was referring to "professionals" that make this for a living, not enthusiasts that make tests in a forum.

In any case though, in ANY test, there is a level of bias. With enthusiasts, simply the level of bias can only be traced to involuntary bias.


You can google more about "test bias" and see more about it. It is proved that even the preconcept of a tester can influence the result, even if he isn't the one who decided how to conduct a test. (this is why pharmaceutical companies use "blind" and "double blind tests", where the tester or both the tester and the tested subject don't know what they are testing). The way to conduct a test, has itself a bias. Why choose tests 1,2,3 and not 1,2, and 4 which is used by someone other.

This is also interesting to see:
http://www.bbc.co.uk/science/horizon/2002/homeopathy.shtml

For the history, the result was that there was no proof of homeopathy working. Despite what other testers have found previously. There's also one such tester in the documentary who couldn't believe her eyes. The explanation is that her bias (she wanted to prove that homeopathy works), influenced the test and she got "proof" that it was working. While her colleauges in the BBC test, who didn't know what they were doing, proved otherwise.

The more evident bias finally is in the final report. Two different people, getting the same test results, may produce a final conclusion completely different, based on what they think as more important. (which is more or less obvious in Matousec, who sets a level for very good, a level for good, for any type of product, while someone else may consider some tests more important than others or set different limits or don't rank products in the same ranking or put different value of tests for Comodo compared to Mamutu. Meaning, one may expect Mamutu to perform differently in different tests). Another "Matousec" can come up with other tests that can "change" ranking, because they target more the vulnerabilities of a product or... surprise, surprise, don't focus so much on HIPS but on packet filtering! What's the "safest" firewall at the end? The one who passes more leak tests? The one with best inbound protection? The one with SPI? The one with ARP? Is the "Wallbreaker" as important as the "PC Audit" test or in real life their importance should be different? How many leak tests should be considered in a ranking? 10? 20? 30? What's the level of "best" , "good"? Is it still "recommended" firewall even if it causes your connection a 20% slowdown? Is it "recommdended" even if it eats your CPU cycles for fun?

Similar things happen with AV tests. The same av scored 99% in one test and 85% in another. The answer lies in a sampled bias and possible unconscious testers' bias.

OFF TOPIC START

This is the latest on my list. I have just ordered this Hitachi Hard drive (well it's "B" version, which is pretty much the same). Although "testers" measure more or less the same things, it's "mysterious" how they "see" different things. Just some samples, you can find differencies in every test in the internet, while they use the same software and hardware usually!

Access Time, different programs, give very different results:

13.8 ms in Tom's hardware. Clearly the drive sucks.
http://www.tomshardware.com/reviews/hitachi-western-digital-terabyte,2017-5.html


12.55 in Xbit labs (one of the best access times specially if you test read too!!!)
http://www.xbitlabs.com/articles/storage/display/1tb-14hdd-roundup_10.html#sect0


Tom's hardware: PC Mark 2005 test (file write)
76,1 MB/s
http://www.tomshardware.com/reviews/hitachi-western-digital-terabyte,2017-7.html

XBit labs: PC Mark 2005 test (file write):
84.08 MB/s
http://www.xbitlabs.com/articles/storage/display/1tb-14hdd-roundup_18.html#sect0

The best comes in the "conclusions" (like in "what's safest"):

In Xbit labs, the Hitachi takes the Editor's choice (reccommended by "Matousec A"):

"Hitachi E7K1000 and Western Digital Caviar Black with our Editor's Choice title as the best HDD choice for a performance home computer system"
http://www.xbitlabs.com/articles/storage/display/1tb-14hdd-roundup_22.html


In Tom's Hardware, it's not reccommended for performance, but for storage (not reccommended by "Matousec B"):

Hitachi's new Deskstar 7K1000 is clearly worth recommending for those with huge storage requirements. Its 1 TB represents a 33% increase over Seagate's 750 GB, which is substantial, and it even outperforms the Seagate Barracuda 7200.10. However, it is not the perfect hard drive at all. Performance users should still go for a WD Raptor drive at 10,000 RPM to host the operating system, and add a second 7,200 RPM for storage. In such a case, the 7K1000 is awesome, but it also gets noticeably hotter than other 7,200 RPM drives, so it should be properly cooled for the sake of data safety.

http://www.tomshardware.com/reviews/hitachi-7k1000-terabyte-hard-drive,1584-9.html


Then the funniest part comes to the review of the same HD but "B" version. The summary is oustanding:

While "All Deskstar 7K1000.B drives spin at the standard 7,200 RPM speed, but they are much more efficient than their predecessors as a result of the decreased platter count."

we have:

"this is as fast as a 5,400 RPM notebook hard drive and hence not very impressive at all."

http://www.tomshardware.com/reviews/hitachi-western-digital-terabyte,2017-2.html

How odd. It's an evoluted version of the previous Hitachi, but now it's as fast as a 5400 rpm notebook harddrive...

Also interestingly, for the "B" model:
Tom's hardware gives: Access time, 17.1 ms
http://www.tomshardware.com/reviews/hitachi-western-digital-terabyte,2017-5.html
Xbit Labs, gives: Access Time: 12.62 ms
http://www.xbitlabs.com/articles/storage/display/1tb-14hdd-roundup_10.html#sect0

That's quite a difference... One got the miracle disk and the other got the lazy disk?


And you should see how different chipset tests are and even worse the conclusions! In one review the AMD 970gx chipset was consistently beating the nv8300 and the conclusion was the the "nv was best for games, while the AMD for home cinema".

OFF TOPIC END

I think you are seeing personal attacks everywhere, while you shouldn't. Well, unless you 're Matousec... (it's a joke). If this can make you feel any better, i exclude you or other amateur testers from what i said.

 

NO! Never have. Never will.
Matousec's opinions are of no value and the site serves no useful purpose.

 

Why would I trust someone that include a Behaviour Blocker (Mamutu) in Firewall tests?

 

Matousec's tests, have a value. On seeing how firewalls fare against a specific threat, called leak tests and in particular against the leak tests , which Matousec chooses to use in his test array. That's all. This is only one part of what a firewall should be reccommended for. And certainly, the "recommended" should be towards expert users. Because i can't give the Comodo with D+ to a novice firewall user and tell him "use that, it's reccommended".


As for including Mamutu, TF and the likes, that's like doing a ranking for "best fruit" and include in the same ranking, fruits with the best taste, fruits which are best for satisfying your thirst in summer, fruits that are conserved longer and test them against a series of tests which concentrate on sweetness. Of course water melons, which are perfect for summer thirst and relief, will rank below figs (being full of sugars) and so on...


Take for example a firewall that ranks "medium" in Matousec's leak tests, because has a medium strength classical hips module. If it encounters a malware which uses a way that bypasses it, it's useless. Mamutu may fail on individual leak tests. But, it may indeed STOP the same malware just because, it's NOT a classical HIPS. Hence action 1 may not trigger it, but action 1 + action 2, may be a behaviour trigger and thus actually STOP the malware. This of course, isn't tested in Matousec (as live malware isn't tested generally). And what's more important, a behaviour blocker may be recommended to novice users, whereas one with classical hips may not be.

I 've run some dozens of live malware under Shadow Defender and Sandboxie and apart the fact that most of them trigger even the simplest firewall with no leak protection (reality vs POC importance), most malware will do other things than just try to make a POC to phone home. A 40% of those that i tested trigger registry changes. That alone may be enough for Mamutu or any behaviour blocker to catch it, just like Twister's FDDS does.

And what if a firewall is superstrong in HIPS, but has holes in its packet filtering? Is it still reccommended?


Every test has its value, but must be seen critically. And the best of all, is think what your own conclusions are before reading the tester's conclusions. Results is one thing. The way you report the results is a totally different thing and part of the tester's bias.

 

Same here.
I installed Jetico firewall and wanted to use it but it killed my internet connection. Go figure it out. It never happen to any FW ever as i far as i remember.

 

Hi Mike,

I agree with you.
I know that this site was promoted by Comodo when they didn't scored well on Matousec (An Irony).

Don't worry about, we don't take them as "holy grail" results.

Good remark.
Don't worry it happen to all of us that are willing to help others (some helping online and great as you, and some like us, helping to our friends, neighbours, relatives, etc)

It's not for laugh(yes i know that), but where i live all the discussions are ending this way. (And this is for grown and adult people).

 

UK comodo moved CIS into spyware freebie for its own sole purposes - who knows. UK now moved its assez from Matousec site for its own sole purposes. UK is the most close tie to U.S. always-on internet lurking, backdoored softwares like in MS' products suspicious. FBI admitted recently wide spread its own lurkers malicious wares into people computers.

Now, Matousec is owned by a private company in U.S. That could be a grain of salts. It is still looking good for its open source codes to all of its proactive security suite test toolkits.

 

It is now "Proactive Security Challenge". Did Mamutu went to perform better after the name of the challenge has changed ?

The names can be misleading. You need to read everything carefully, not just the names, before falling into the conclusions. When it was "Firewall challenge" the tests were the same, they actually were about proactive security.

 

Yes, I do. Considering how full of crap Matousec is, I make sure to stay the hell away from whatever they recommend.

 

The better way is to stay away from ANY recommendation and to test everything by yourself. In the end you will be surprised how full of crap your results are

 

Been there, done that since years ago. What I am surprised at are how full of crap some people can be.

 

Oh, 'cmon, share your results with us. THEN you'll find what I have promised. Do not try to find it yourself, let others do it

 

Off topic

Because of what was said in this topic Mike and I talked, and I think we understand each other now

Greetz, Red.

 

".... full of crap Matousec is..."


Pray tell, how so??


|||

 

A firewall that scores bad at matusec is out of the picture for me..

No reason not to score okay there.

 

Clearly Matousec scores "firewalls" not as pure firewalls; he scores them as HIPS/firewalls. If they do not protect against unwanted/dangerous internal process activations, then they are scored badly; but that is not what a pure firewall that controls only the incoming connections allowed into a network does.


|||

 

I think their tests are good and valid, an application that scores well there has good protection against what they tests are testing for.

My main issue is the way they show the results and rank the applications based on it. They test keylogger and not IPV6. Then how do you accurately rank the firewalls, on these results.

Something like threat firewall should be tested against keyloggers, kill tests, etc. It should not be tested with do they control outbound protection.

 

So, who will create the "perfect test/evaluation"?

The question of the ages.


|||

 

Yes, fuzzy marketing of Comodo, also the independant illusion of not letting Comodo score the full 100% is smart.

 

Actually this is done with almost all the FW vendors.
When Comodo scored well it was promoted big. When it was going down they said that testing shouldn't be trusted so much.
Same thing happen with one of the other products( OA or OP).

Same with NOD32. When they scored well in av-comparatives, that was shown on their page as main argument against other products.

No disregard to Any company(Fw or Av), but they are using the results as they want. If they are good in testings, they praise them self for of couple years or some period of time, and after that if results are bad they are trying to minimize the effect of the Same testings. And Vice Versa, if the results weren't good to another company they are saying that the test aren't reliable, and when the moment comes that they be in the top they are saying that are best and that is show in the results.

 

This kind of comment just makes my jaw drop honestly.... With no thought of doing research or investigation for your self.

As for me Matusec has no baring on my choice of firewalls the only thing that site is good for is a Occasional laugh.

 

It also comes to show, how feasible it is to make a "transparent" test, that can put some distance between product A and B (specially if you want to). I mean, most probably, the tests of that site (the one sponsored by Comodo) are real. By real, i mean that for the versions referred, they don't lie on the results and Comodo was ahead.

This simply is yet another proof of how the tester can arrange by selecting which tests to use, the final results. Since there is no worldwide test criteria and no official title of "professional tester" (there is no university diploma to make "pro tester), each tester can choose the battery of tests and test subjects that most likes. So, in the Comodo sponsored site, the tester, had no trouble in showing that "Comodo is the best" (surprise, surprise). In Matousec, a different battery of tests in used and results are different. Also non firewalls are included, because the tester likes so.


So tests show something in general , but if you want the absolute truth... After a point as Renegade said, it's all about marketing. It's about getting the "logo" for show on your website.

Another tester could disregard completely the leak tests and concentrate on packet filtering for example, may come with completely different rankings.

Another tester could use REAL malware instead of leak tests and give completely different results, as real malware, usually does various things, that most likely would trigger more software, reducing the distance between the top dogs and the rest of the bunch.

With AVs the differentiation can be much wider. What percentage of each malware type to inclulde, geographical origin prevalence, frequency in real world of the malware....

 

I wonder... If i asked Matousec to include Twister, would he do it? It does have a behaviour blocker, so if Mamutu and Threatfire are eligible, why not Twister??

BTW, why hasn't he still tested Norton's AntiBot?

 

On a Off note diffident Matousec just get bought or or some crap like that I thought I remembered seeing something about that a month or so ago.

 

Yes, someone bought Matousec and said that the test will continue. I do find a bit weird that someone is paying money (in the middle of an economic crisis) to get a testing site that apparently doesn't generate any income. Apparently being the key word. Maybe "independent testing" has indirect ways of generating income... Because buying for charity reasons is something unheard of. Even Melih with all the "let's save the world from malware" rhetoric, at the end found a way to generate a revenue from additional services and Ask toolbar.


The only trully "independent" testing i know of, is from enthusiasts made in fora. When you see $ involved, there is something fishy behind.

For example, if i were an investor and wanted to buy Matousec, i would pay my precious money, but, i would make sure that i would get back the my money and more, from activities related to the testing.