Why Does MATOUSEC Rate "Smart Security" So Low?

[Dennis7]

Can someone please explain to me why the so-called respected firewall security testing site www.matousec.com rates ESET SMART SECURITY second from the bottom? How can matousec claim that Smart Security offers NO PROTECTION LEVEL with a 4% Level 1 product score, thus earning a NOT RECOMMENDED overall evaluation???

Please scroll down the link and see for yourself!
http://www.matousec.com/projects/proactive-security-challenge/results.php

I am currently on a 30 day free trial of ESET SMART SECURITY, and so far it is working far, far better than any other firewall I have used in the past!!!

In the last month, I had to give up on the two top rated Matousec firewalls, ON-LINE ARMOR and OUTPOST SECURITY SUITE PRO for the following reasons:

OUTPOST SECURITY SUITE: Failed to prevent infection of a common trojan horse program and only detected 3 of 7 malware registry changes. The routine scan picked up the 3 registry entries but failed to detect the .exe trojan horse that planted a WINFX folder to hide in under c: program files.
I had to finally give up on this program when after a few weeks for some unknown reason it would not allow me to download from certain sites. I even did a clean uninstall and re-installed a just released newer version. SAME PROBLEM AGAIN! Also, at the time auto-updates would not indicated when a newer version of the program was available, let alone perform the download/install by itself.

ON-LINE ARMOR: No issues with malware. This program only lasted a few days on my computer. I got tired of getting pop-ups from TRUSTED programs that ON-LINE kept asking me if I wanted to allow them to run. A ton of balloon pop-ups concerning other computers connecting to my wireless network in my house. The major items were when this program totally messed up my download/install of the new Internet Explorer. I think it was firewall permission issues and interfering pop-ups. It messed up my current version of internet explorer so badly that I had to do a RESTORE and download the new version of IE all over again at microsoft update. Then I had to download another program so I decided to EXIT ON-LINE ARMOR so it would not have a chance to interfere with the download/install of it. I enabled windows firewall to have some firewall protection during the download. Well, once I exited ON-LINE ARMOR it again totally destroyed my main browser FIREFOX. I had to again download and install FIREFOX all over again. And it also completely disabled my internet connection! For some reason ON-LINE ARMOR would not allow my ISP to establish a host address with my computer to enable a connection. This problem was solved after I uninstalled ON-LINE ARMOR.

Last year I had the misfortune of buying KASPERSKY anti-virus at BEST BUY before any trial period. I WILL NEVER DO THAT AGAIN! I will have to try a product before I buy it!! KASPERSKY had by far the most pop-ups and warnings of any program I have ever used! It cried wolf on almost every program that launched!!! I think you needed a PHD in computer science to understand the user-interface of the program!

It's been a long time since I used NORTON. NORTON would always cause major instability issues with my system. But I am tempted very much to give the 2009 or 2010 version of their security suite a try some day.

Before all of this mess, I was humming along very nicely with the combo of Zone Alarm Pro firewall and NOD32 antivirus. Then I kept reading those STUPID reviews and decided to switch to the so-called TOP RATED security products. My Zone Alarm subscription ran out, I still had till March 2010 with NOD32, but I decided to experiment and test other security programs.

Well, as mentioned above, I am FULLY and COMPLETELY back with NOD32. This time I am trying out the Smart Security Suite. I am very pleased so far!

When I first installed ESET SMART SECURITY, I ran it through every test on both the SHIELDS UP and PC FLANK security testing sites. It passed with a perfect score of 100 %!!! All ports were stealthed. No leaks. No etc. etc. etc. I ran it through every test I could find on those sites. No exploits or weaknesses were ever found!

What good is a so-called TOP RATED software security program if it makes your computer system unstable, problematic, and pesters you with pop-up after pop-up!

The next testing phase of ESET SMART SECURITY for me will be if it can remain stable and play nicely with my computer for the remaining trial period.
Then the final testing phase will be on day 29 of the trial when I do a full deep scan of my computer with SMART SECURITY. Then I do follow-up scans with the free Google NORTON on-demand scanner, the free AVIRA anti-virus on-demand scan (YES I know you can not have two anti-virus scanners running at the same time....but you can disable the back-up antivirus and only enable it after you disable ESET....thus you only have one running at the same time and the other disabled on stand-by to get second and third opinions). And I also have the free Google SPYWARE DOCTOR to run a deep full scan for spyware/adware/malware.

I will then examine what the other scanners pick up that ESET missed.

I will do some other posts very soon concerning Internet Security Suites!

Hope this info helps!

[Najem1992]

hi

in answer to you question .... those people rate security programs according to the sum of money given to them by the company therefore because Eset didn't bribe ( give moeny to ) them they put it at the bottom

that's my opinion ;D

and thanks so much for the info
they are really great

[stackz]

It's been explained before, but basically the matousec tests are HIPS (host intrusion prevention system) tests. ESS anti-virus is disabled for the test so no chance of heuristic detection. ESS does not have HIPS (at least as far as the HIPS implementations used by the top scorers goes) and as long as it doesn't and the testing methodology remains the same, I'd expect similar results. Seriously if you're going test a security product, then test the entire product - not with half of it disabled. :wacko:

[Najem1992]

-{ Quote: "It's been explained before, but basically the matousec tests are HIPS (host intrusion prevention system) tests. ESS anti-virus is disabled for the test so no chance of heuristic detection. ESS does not have HIPS (at least as far as the HIPS implementations used by the top scorers goes) and as long as it doesn't and the testing methodology remains the same, I'd expect similar results. Seriously if you're going test a security product, then test the entire product - not with half of it disabled. :wacko:" }-
thanks for explaining
and I hope Eset would put HIPS soon ::)

[osip]

I totally agree with Dennis7, although I have not experienced the kind of problems he describes with other products ...Personally I welcome ESET with their latest v4 version. Combine with sandboxie/prevx,zemana or Threatfire and you will be top protected enough...

[Kees1958]

Answer is as simple as this

Smart security will look at it at a smart way, meaning a sequence of events (e.g. executing a file from temp, stopping a service, setting a file overwrite with pending file rename operations of that original service, bingo on next boot).

Matousec looks at single intrusions. Smart will most likely ignore most single intrusions (unless they involve a direct ring-0 attack). So that explains the low scores.

Regards Kees

[Dennis7]

Thank you all for your replies!

I definitely agree that money/bribery/"payola" has something to do with reviews in all product areas, not just computer software. Advertisers support the magazine or website. I am sure pleasing them has a much greater priority then just being honest to the public. I have seen so many computer websites that rate some software product I never heard of as number one in it's category. An obvious indication that advertising dollars were at work here!

I have two questions in regards to MATOUSEC testing SMART SECURITY:

1. Then why is the ANTI-VIRUS disabled? If this lowers the test performance rating score, I see no reason to have it disabled.

2. Then if the best products tested have HIPS.....then why don't ESET SMART SECURITY have it???

Please advise me on these two questions, Thank you!

[Someone]

-{ Quote: "in answer to you question .... those people rate security programs according to the sum of money given to them by the company therefore because Eset didn't bribe ( give moeny to ) them they put it at the bottom

that's my opinion ;D" }-
Do you have any proof to backup this statement?

[Franklin]

Computerworld Article (http://www.computerworld.com/action/article.do?command=printArticleBasic&taxonomyName=Security&articleId=9132568&taxonomyId=17)
-{ Quote: "The publication's June issue rated 11 commercial security suites, including high-profile products like Symantec's Norton Internet Security, McAfee's Internet Security and Eset LLC's Smart Security. Each package was tested against both actual exploits active on the Web as well as others that were modified to represent "threats too new to have been discovered," said the magazine.

Eset's Smart Security led the list with a score of 72 out of a possible 100, with McAfee's and Symantec's suites in second and tied for third, respectively, with 71 and 70 points. (Microsoft Corp.'s Live OneCare, which will be discontinued June 30, tied Norton with 70.)" }-

[NoIos]

Consider this...those that are in the top positions...have great products yes...no doubt..but believe me, they do their homework well. They know what tests matousec does and prepare their products accordingly. This is why one month you'll see Comodo first...the other month is the turn of Online Armor etc.
Not all companies care to do this or care to do this constantly. Actually I believe that the companies that are into this race are those who sell less enterprise products and have as main client base what it's called...home users.

For the rest others have covered me here.

[tipo]

-{ Quote: "hi

in answer to you question .... those people rate security programs according to the sum of money given to them by the company therefore because Eset didn't bribe ( give moeny to ) them they put it at the bottom

that's my opinion ;D

and thanks so much for the info
they are really great" }-
beautifull :thumb: :thumb: :thumb:

[tipstir]

All about money today.. Can't trust everything you read in a review.. Best to test the software out yourself..

[Dennis7]

Thank you all again for your replies! Please keep them coming!

I am still waiting for the answers to my 2 questions in a previous reply post:

1. Then why don't SMART SECURITY enable the Anti-Virus in the MATOUSEC testing, or at least demand MATOUSEC allow them to, in order to test the complete product fairly???

2. Then if the top-rated products have HIPS, then why don't SMART SECURITY have it, or at least plan to include it in the near future?

(By the way, what exactly is "HIPS")

To all those that desire proof of "Money talking" in software reviews....

I remember an episode that "60 Minutes" did several years ago. There was this segment on how automobile executives would pay for vacations, meals, hotels, cash spending money, etc. to reviewers of major publications that rate automobiles. If it happens with cars, it happens with OTHER PRODUCTS as well!

So many people have complained why PC MAGAZINE would almost always give NORTON INTERNET SECURITY a "EDITORS CHOICE" Award review, even when this product was unstable, a resource hog, and buggy! You can always go to unbiased review sites like cnet.com and download.com and read the user reviews and notice that NORTON would achieve an average score of 1.5 stars out of 5! And yet they would keep winning EDITORS CHOICE by PC MAGAZINE! ----Oh by the way, did you notice how many full page ads NORTON would buy at PC MAGAZINE???

And again, I want to mention how many times I went to a web site that reviewed software products, and noticed the number one rated product in a particular category was some product I never heard of! ----Advertising dollars definitely at work here!

[dawgg]

-{ Quote: "in answer to you question .... those people rate security programs according to the sum of money given to them by the company therefore because Eset didn't bribe ( give moeny to ) them they put it at the bottom

that's my opinion ;D" }-
(In my opinion), your comment is totally wrong, you have no evidence of this and can not justify that all other companies have been "bribed" by Matousec. Do not make comments you know nothing about and damaging the reputation of others with no justification, only to make your favorite AV look better than the others. All other AVs do not necessarily score better than your favorite because *maybe* they paid a "bribe" - maybe, just maybe they are simply better at the leaktests.

Matousec if quiet transparent with its testing and everyone is easily able to recreate the tests and should attain very similar results to that which Matousec gave.

http://www.matousec.com/projects/proactive-security-challenge/level.php?num=1
http://www.matousec.com/projects/security-software-testing-suite/
http://www.matousec.com/projects/proactive-security-challenge/results.php#detailed-results

[alex_s]

-{ Quote: "Answer is as simple as this

Smart security will look at it at a smart way, meaning a sequence of events (e.g. executing a file from temp, stopping a service, setting a file overwrite with pending file rename operations of that original service, bingo on next boot).

Matousec looks at single intrusions. Smart will most likely ignore most single intrusions (unless they involve a direct ring-0 attack). So that explains the low scores.

Regards Kees" }-

The problem is a sequence of the events is theoretically unpredictable, so trying to do smart you risk to miss a danger. It just may be too late. From ESET results I see that it failed kill1 and kill2 tests. My personal opinion is killtests are the "must-pass" tests for any security. If a security can be killed from a usermode this is a very bad sign.

[Kees1958]

-{ Quote: "The problem is a sequence of the events is theoretically unpredictable, so trying to do smart you risk to miss a danger. It just may be too late. From ESET results I see that it failed kill1 and kill2 tests. My personal opinion is killtests are the "must-pass" tests for any security. If a security can be killed from a usermode this is a very bad sign." }-

ThreatFire allready creates a log of events after the first intrusion. In a private e-mail with PrevX they told me that they are considering a virtualisation extention on the allready existing heuristics tracking mechanisme. Comodo announced a simular time travel feature. My guess is that behavioral blocking/heuristics analysis will evolve to virtualisation also.

It would applied using (old) existing data base backup/recovery mechanisme. When a multiple data transaction starts you set a restore point. While doing updates, you have to commit your changes when the last transaction succesfully finished. When an erro occurs, along the way, you simply roll back to the restore point.

For HIPS it would be easy to develop a simular mechanism. When the first intrusion occurs a restore point is set, then it starts tracking the flow of events. When the heuristics/behavioal analysis decides it is enough it pops-up. When the user decides to block, they roll back to the created restore point, otherwise they apply a commit, transferring registry and data changes from the sandbox to the real file system.

With the above observation, I do agree with your post, but it will be a matter of time. Windows 7 even had such a mechanisme build in, maybe they with drawn it, to implement it in their successor of windows security.

Regards Kees

[Dennis7]

dawgg,
I have no evidence if MATOUSEC accepts any form of bribery, but you can certainly believe that it DOES happen in many other review sites/publications, per the examples in my last reply post.

Well then again, maybe this paragraph taken from the MATOUSEC site can be a form of "special treatment" if a vendor is willing to pay for it:

The following quote can be found at: http://www.matousec.com/projects/proactive-security-challenge/
(Scroll down to the second paragraph in "Product Selection and Vendor Rights".)

************************************************
Paid testing

Every vendor has a right to request a paid Proactive Security Challenge testing, in which case its product will be tested in all levels regardless the results on each of the levels. After the vendor receives the results of the paid testing, it can either keep them private or request their publishing on our website, but such a request will be satisfied only if the previously published results for the tested product, if any, are at least one month old and if the tested version is stable and publicly available. There are no limits of the frequency of the paid tests
*************************************************

Hmmmmmmmmmm, why does a so-called fair unbiased testing site like MATOUSEC have to offer "special treatment" in their product evaluation testing to all those vendors willing to pay a price for it???
Obviously MATOUSEC is profit motivated! They are willing to modify the results and make them look prettier if a vendor is willing to fork over some cash! I say that MATOUSEC is much more interested in making a profit then giving a fair ranking to tested products! After all, we are testing the overall security of a product---not how many times a vendor is willing to pay for "repeated testing" and/or "special testing provisions".

So if MATOUSEC tests/evaluations are genuine and a correct indication of a firewall and/or security suites performance....then why don't everyone immediately uninstall ESET SMART SECURITY, cancel all plans to renew subscription, and just go ahead and buy a firewall/security suite that is rated no lower than the TOP 3 at MATOUSEC???

I strongly suggest that someone associated with ESET SMART SECURITY reply to this post and offer some adequate defense for SMART SECURITY's very low ranking in MATOUSEC testing!!!

And of course, if there is anyone associated with MATOUSEC in this forum audience, I also welcome your reply as well!

Currently, when my 30 day trial is up with SMART SECURITY, I plan on going with the 2 week free trial of NORTON SECURITY 2009 followed up by the 30 day free trial of KASPERSKY SECURITY 2009. If any of these two suites play nicely with my computer, giving me no problems, then I may just go ahead and decide to purchess a one years license from either one. I checked out the links provided by dawgg, I find ESET SMART SECURITY's firewall performance grossly unacceptable, and I encourage all readers of this thread to likewise at least strongly considering a test drive of a much rated higher firewall/security suite!!!

[osip]

I-{ Quote: "I strongly suggest that someone associated with ESET SMART SECURITY reply to this post and offer some adequate defense for SMART SECURITY's very low ranking in MATOUSEC testing!!!" }-
-{ Quote: "I find ESET SMART SECURITY's firewall performance grossly unacceptable, and I encourage all readers of this thread to likewise at least strongly considering a test drive of a much rated higher firewall/security suite!!!" }-

So, you agree with Matousec then...