Anti-Malware.ru: (Zero-Day) malware test

[progress]

Read more ... (http://translate.google.at/translate?hl=en&sl=ru&u=http://www.anti-malware.ru/antivirus_test_zero-day_protection) :)

[andyman35]

Interesting test but given the extended period over which the tests were performed some were tested using older versions than others (latest DW,older NIS2009 + CIS 3.9 for example).I'm not sure how much this influences the overall results.???

[pandlouk]

-{ Quote: "protected= Malware found explicitly found and defused, or suspicious activity, and the infection completely suspended if the correct choice of the user in a dialog box (for example, revealed a dangerous operation, thwarted the attempt of infection, detected an attempt to run malicious, zabrokirovana network activity detected an attempt to change the files -- Displays a dialog box and warning that most likely triggered by an application is malware)." }- (http://translate.googleusercontent.com/translate_c?hl=en&sl=ru&u=http://www.anti-malware.ru/files/dynamic_test_results2.xls&rurl=translate.google.at&usg=ALkJrhgnfDhD49TISG3-AMTy817fqlHk0Q)
Leaving the choise to the user means failed/infected to me.

Panagiotis

[jmonge]

DefenseWall was the best in the test;) yay

[Creer]

Congrats Ilya :)

[JamesFrance]

The interesting thing about this test is to see how ineffective most security suites are against recent malware. With more circulating every day they need to do much better, especially when they expect payment.

[funkydude]

I like the idea behind this test, but they are going about it wrong. You just can't test HIPS vs AV, HIPS will always win. Hence why everything here that has a HIPS module scores high (Comodo scores high but it's AV is very poor for example).

I can't really think of anything better right now though, but I'm sure someone will put forward a splendid idea.

[jmonge]

-{ Quote: "I like the idea behind this test, but they are going about it wrong. You just can't test HIPS vs AV, HIPS will always win. Hence why everything here that has a HIPS module scores high (Comodo scores high but it's AV it completely useless).

I can't really think of anything better right now though, but I'm sure someone will put forward a splendid idea." }-
dont forget that couple of the antivirus tested has a hips modules,like kav,comodo etc and i think it is fair to test them this way for the test is proactive defense and yes hips will win most of the time:thumb:

[kasperking]

-{ Quote: " but they are going about it wrong. You just can't test HIPS vs AV, HIPS will always win. Hence why everything here that has a HIPS module scores high (Comodo scores high but it's AV it completely useless).
" }-
ummm........its more of "security software" tested against malware....so defense wall/Kis/Cis/avast/avira all fit the bill....means tomorrow defense wall may have an av-component... Internet suites/avs have sandboxing implemented...so the lines are very blurry and getting blurred by the moment.So if a product claims to protect its claims have to be tested and tests are relative :-\

[funkydude]

-{ Quote: "ummm........its more of "security software" tested against malware...." }-

Yeah it's more of a security software test rather than an AV test, I see your point.

[Ed_H]

Ilya Rocks!!!

[ratchet]

So I can't tell from the DW website cost and the difference between paid and free! Help please! Thank You!

[Ed_H]

There is no free version, only paid.

[icr]

Well it is good to see that Kaspersky is again back among top positions because they were bashed for underrated performance at av-comparatives.:thumb:

But still can't figure out how Avira and GData scored badly in detections???

Anyways congratulations to DefenseWall.

[demoneye]

this test may mislead ppl to think DW is some sort of av!
so it isnt , and the person who test avs with DW made a big joke of him self , its like test malware under SB or even in RB environment (and than rollback and say RB pass the test) and compare to any avs out there hehe :argh:

cheers

[maymoons]

-{ Quote: "But still can't figure out how Avira and GData scored badly in detections " }-

Me too. Especially GData
Gdata=avast+bitdefender (without B-have)

[LaserWraith]

I think combining AVs and HIPS and virtualization software makes an interesting test. Of course, products with just an AV will not do as good (duh).

But then, it isn't just an AV test. I like "real world" tests. Testing the defaults of different security suites seems good to me, as it can guide "newbies" to which programs are worthwhile.

-{ Quote: "I like the idea behind this test, but they are going about it wrong. You just can't test HIPS vs AV, HIPS will always win. Hence why everything here that has a HIPS module scores high (Comodo scores high but it's AV it completely useless).

I can't really think of anything better right now though, but I'm sure someone will put forward a splendid idea." }-

CIS's AV may not be the best, but I think saying that it is useless is a bit much.

[funkydude]

-{ Quote: "
CIS's AV may not be the best, but I think saying that it is useless is a bit much." }-

I modified my post. :)

[JamesFrance]

Oh the usual Comodo bashers always creep out of the woodwork when anything shows it to be good at something.

Just ignore it, it's the usual thing on the forum trying to discredit Comodo.

Sooner or later they will get tired of doing it.

[LaserWraith]

-{ Quote: "I modified my post. :)" }-


Thanks. :P


BTW, I don't use the AV anyway. I don't use any (lol).

@JamesFrance: I know many here don't like Comodo...but I feel like that doesn't make most of them "bad". :lurking:

[JamesFrance]

-{ Quote: "


@JamesFrance: I know many here don't like Comodo...but I feel like that doesn't make most of them "bad". :lurking:" }-

Oh I know it's just a few, but doesn't it get boring when they go on and on?

[funkydude]

-{ Quote: "Oh I know it's just a few, but doesn't it get boring when they go on and on?" }-

Bit like you are now? *sigh*

[LaserWraith]

You seem to be both going on and on. :P

[funkydude]

-{ Quote: "You seem to be both going on and on. :P" }-

Complainers.
Complainers who complain about complainers complaining.
Complainers who complain about the complaining complainers of complainers.

See? Even I'm confused. Brilliant life isn't it?

[firzen771]

-{ Quote: "Well it is good to see that Kaspersky is again back among top positions because they were bashed for underrated performance at av-comparatives.:thumb:

But still can't figure out how Avira and GData scored badly in detections???

Anyways congratulations to DefenseWall." }-

it scored badly on AV-C because as its name says its an ANTIVIRUS comparative, this test compares all components.

[aigle]

This test is a joke? They collected a pile of applications including HIPS, AVs, Suites, Sandboxes and tested them altogether. Camparing oranges with apples.

[subset]

-{ Quote: "This test is a joke? They collected a pile of applications including HIPS, AVs, Suites, Sandboxes and tested them altogether. Camparing oranges with apples." }-
Why should a user care if Malware is stopped by apples or oranges. Either security software is effective or it is not.
BTW this sounds like a postulation to test only suites vs suites and sandboxes vs sandboxes etc.
But why? Just to let suites shine in their own little test?

The only thing I don't like is that both added "HIPS" are from Russia. It would have been really interesting to see the results of apps like Geswall, Sandboxie or Malware Defender as well.

Cheers

[SIR****TMG]

:thumb: Way to go Ilya. Thats why I use Defense Wall.:thumb:

[Soujirou]

Is there a reason some (all?) testers stop at a service pack update but don't install any further updates? I can see why they would want to keep the system constant throughout the testing period, but I would imagine that there were updates between SP3 being released and July.

Also the translation may be off but is the report saying it tested 36 links total during the testing period? Are dynamic tests by their nature limited to sample sizes of this range?

[subset]

-{ Quote: "
Also the translation may be off but is the report saying it tested 36 links total during the testing period? Are dynamic tests by their nature limited to sample sizes of this range?" }-
There is a site with the "Test Methodology":
http://translate.google.com/translate?hl=en&sl=ru&u=http://www.anti-malware.ru/node/1922
Seems like it was rather difficult to find Malware which was not detected by more than 20% of the tested Scanners at VirusTotal.

Cheers

[vijayind]

Comodo can stop over 80% and Outpost barely 40% ....
I thought Comodo and Outpost has nearly similar HIPS capabilities.

[pjb024]

-{ Quote: "Comodo can stop over 80% and Outpost barely 40% ....
I thought Comodo and Outpost has nearly similar HIPS capabilities." }-

The report says that all AV's (their wording even though not all contenders were AV's) were tested with standard default settings which, for OSS, could mean it was in learning mode. Certainly OSS default settings won't pass leak tests and is not a fair reflection of the capability of the suite. In my opinion tests like this are of little value other than curiosity. There is too little information on the setup of each contender.

By the way, when I click the link to that site to view the report I get many suspicious packets blocked by OP (I'm currently using the FW not the full suite but have licences for both and have used both).

[subset]

-{ Quote: "
By the way, when I click the link to that site to view the report I get many suspicious packets blocked by OP" }-
With the Google translate link? :lurking:

Cheers

[Lebowsky]

-{ Quote: "I like the idea behind this test, but they are going about it wrong. You just can't test HIPS vs AV, HIPS will always win. " }-
;D yeah, i just wish more people knew that.

[Lebowsky]

-{ Quote: "Why should a user care if Malware is stopped by apples or oranges. Either security software is effective or it is not." }-
Agreed.

[hamzah95]

Good to know that I'm using the Best protection.

.........
..........
.......
DefenseWall:thumb: :thumb: :thumb: :thumb:

[jmonge]

-{ Quote: "Good to know that I'm using the Best protection.

.........
..........
.......
DefenseWall:thumb: :thumb: :thumb: :thumb:" }-and you dont have to be waiting for database updates;)

[pjb024]

-{ Quote: "With the Google translate link? :lurking:

Cheers" }-

Yes that's correct.

[pjb024]

-{ Quote: "and you dont have to be waiting for database updates;)" }-

I'm not going to bash DW because I happen to think it's good at what it does but I found it too heavy on my system. I know you guys love it because the praise flows freely in almost every thread of the forum. :P

[Brummelchen]

same as aigle - same useless test as matousec offers - only another color.

[JamesFrance]

Hardly a useless test, when it shows how many so-called Security Suites are quite incapable of preventing new malware infection.

Who cares which antivirus can identify the most long dead threats, if they can't stop what is happening now?

[Ilya Rabinovich]

-{ Quote: "but I found it too heavy on my system." }-
Why did you consider DefenseWall as "heavy"?

[Boost]

:thumb: Defensewall

[pjb024]

-{ Quote: "Why did you consider DefenseWall as "heavy"?" }-

I'm not bashing DW, Ilya, as I happen to think it's a unique solution and the concept is great. I tried it a few weeks ago and there was a definite delay when I load my browser (firefox). I am in the habit of closing and reloading my browser often so it became an annoyance. I guess that any virtualization or rules based product would add some kind of overhead as you can't get something for nothing. I'm open minded and when DW3 is the current version then maybe I'll take another look.

I like that you are visible in the forums and quick to respond to user concerns. That's something the big companies seem unable to achieve. Keep up the good work.

[pjb024]

-{ Quote: "Hardly a useless test, when it shows how many so-called Security Suites are quite incapable of preventing new malware infection.

Who cares which antivirus can identify the most long dead threats, if they can't stop what is happening now?" }-

I could write a report on a comprehensive test I have just completed and publish the results along with a link to the report and you would believe whatever I said. If someone wants to pay me to put their product in first place send me a PM :P

We see these so called tests all the time and the methodology employed is usually lacking in any great detail so how can we be sure that the results have any meaning or value other than scare mongering? Do we blindly accept results then go changing our AV if it's shown way down the list? I think that's what happens a lot.

This latest test used 'out of the box' default settings for each of the products tested. That tells me they took that approach because it's the easy way to do it and doesn't involve the testers having to know anything about the product they are testing. We all know that AV's, HIPS, FW etc all need to be configured properly to give a desired level of protection. You need to get to know how each security software works in order to get the best protection from it.

This test is flawed but the gullible will see a table of results and think that many of the products tested are providing very little protection when, in fact, that is not true. You only have to look at the variance in results from different tests to see that far from providing any clarity these tests are simply causing confusion and creating fog.

The only way to get meaningful results is to do your own test.

[Ilya Rabinovich]

-{ Quote: "I tried it a few weeks ago and there was a definite delay when I load my browser (firefox)." }-
OK, that's why I ask. Usually, FireFox slowdown because of DW is about two or three seconds at my computer. Well, four maximum. If, in your case, time delay is more then this numbers, you need contact me in order to initiate investigation process.

[Ilya Rabinovich]

-{ Quote: "see a table of results and think that many of the products tested are providing very little protection when, in fact, that is not true." }-
Unfortunately, it's true. If security software is so strong, why their customers keeps get infected, over and over again?

[jmonge]

forget about the antivirus technology it doesnt keep up their database againts the thousands of new nasty malware,we need proactive protection in our pc's;)
i know some people spend alot of money in security suites and still get infected;D for example i have a friend he called and invite for a diner tomorrow for the purpose of pc clean up and he has avast antivirus and other antispyware solution with a firewall and still got hit;D i bet with just defensewall on his computers i will not be invited for the dinner;D thanks to avast and his antispyware solution i will eat tomorrow lol:argh: ;D

[pjb024]

-{ Quote: "forget about the antivirus technology it doesnt keep up their database againts the thousands of new nasty malware,we need proactive protection in our pc's;)
i know some people spend alot of money in security suites and still get infected;D for example i have a friend he called and invite for a diner tomorrow for the purpose of pc clean up and he has avast antivirus and other antispyware solution with a firewall and still got hit;D i bet with just defensewall on his computers i will not be invited for the dinner;D thanks to avast and his antispyware solution i will eat tomorrow lol:argh: ;D" }-

Did your friend get infected after visiting your website? :shifty:

[JamesFrance]

-{ Quote: "

This latest test used 'out of the box' default settings for each of the products tested. That tells me they took that approach because it's the easy way to do it and doesn't involve the testers having to know anything about the product they are testing. We all know that AV's, HIPS, FW etc all need to be configured properly to give a desired level of protection. You need to get to know how each security software works in order to get the best protection from it.

" }-

No I think you are wrong. I think this is the way to test because any complete security needs to work for someone with no knowledge of computers. It is no good having a default configuration which fails. Most people would have no idea and less interest in altering the default settings. They need protection the most.

[pjb024]

-{ Quote: "OK, that's why I ask. Usually, FireFox slowdown because of DW is about two or three seconds at my computer. Well, four maximum. If, in your case, time delay is more then this numbers, you need contact me in order to initiate investigation process." }-

Yes it was probably three or four seconds. I'll try it again sometime when I have more time to spare.

[pjb024]

-{ Quote: "No I think you are wrong. I think this is the way to test because any complete security needs to work for someone with no knowledge of computers. It is no good having a default configuration which fails. Most people would have no idea and less interest in altering the default settings. They need protection the most." }-

These are the same people who will click on a popup and bypass security anyway. :wacko:

[Windchild]

-{ Quote: "
This latest test used 'out of the box' default settings for each of the products tested. That tells me they took that approach because it's the easy way to do it and doesn't involve the testers having to know anything about the product they are testing. We all know that AV's, HIPS, FW etc all need to be configured properly to give a desired level of protection. You need to get to know how each security software works in order to get the best protection from it." }-
-{ Quote: "Unfortunately, it's true. If security software is so strong, why their customers keeps get infected, over and over again?" }-

I'm going to have to side with Ilya on this one. Regardless of how good or bad this particular test is, real world experience shows every day that people are getting infected in spite of having security software like big-name commercial AVs or popular free AVs or "firewalls" installed and running on their systems. Who here has not seen an infected system that was running an up-to-date Norton suite or something? Fact is, against new malware, such as some trojan-of-the-day that was released 45 minutes ago on a drive-by-download site, most security software is of little use. AVs for example will not have the definitions for such a new malware yet, and if the AV happens to have some kind of HIPS or cloud component to protect against new malware, that still won't necessarily stop social engineering attacks where the user has been fooled to want to run the malware and will just click yes when warned that the file may be up to bad things.

As for default settings, sure, it's typically true that with some configuration security is improved. This is true for many things, first of all Windows, browsers, and even security software. The obvious problem is, how well can the average Joe User perform the configuration changes needed to increase security and how likely is he to even know about the possible need of changing the configuration to increase security? Not very well and not very likely is the answer usually. In real life, a vast mass of users are using everything with the default settings as far as security goes. We are dealing here with the kind of people who have not all yet discovered that there are other fonts than the MS Word default Times New Roman. These people rely on the default settings - if the default settings are not so good, then they are the ones who suffer from it. Compare, for example, how many Linux users run as root (root is not the default) with how many Windows users run as admin (admin is the default)...

-{ Quote: "This test is flawed but the gullible will see a table of results and think that many of the products tested are providing very little protection when, in fact, that is not true. You only have to look at the variance in results from different tests to see that far from providing any clarity these tests are simply causing confusion and creating fog.

The only way to get meaningful results is to do your own test." }-

I certainly believe security software testing in general is not worth much. Most tests are silly one way or another for reasons such as testing against a set of malware samples that is far too small to matter. Typically testing seems to serve only two purposes: 1) getting the testing group some name and 2) getting some folks switching from one security software to another in their endless search for the "best" and possibly shelling out a lot of cash to happy security product vendors in the process.

But I think the idea of only being able to get meaningful results by doing one's own tests is problematic. First, would the kind of users who actually need security software be qualified to test security software in any meaningful way? Can they build a safe testing environment, locate enough samples of malware and exploits, and then actually perform the test? Nah. Which leaves us with the thought that the only people qualified to do any meaningful security software testing wouldn't really need said security software to protect themselves and therefore might not have any reason to even want to test them...

Of course in new malware or exploit testing any kind of security feature not relying on signatures is likely going to be stronger than signature-based products. The problem with these non-signature-based methods is that they either directly require users to make wise decisions ("Unknown process foo.exe wants to perform some fancy technical stuff you don't understand and this may be dangerous, allow or deny?") which tends to be a decently reliable recipe for disaster, or at the least they provide ways for the user to still screw up magnificently unless he knows what he's doing ("Oh, look, this malware I just downloaded in my limited user account is saying that it needs admin rights, so I guess I'll just give the admin password now and see what happens" or "Hey, this program says it doesn't want to run sandboxed and untrusted. I guess I'll just run it in the real system then to get it working for realsies.") which is something that most users don't.

But before I digress further, where does all this leave us?
- The single most important thing in security is knowing what you're doing. If you don't, there's always the pretty real chance that you may get owned by incompetence in spite of having AVs and HIPS software, limited user accounts and what not. Even if you're in a read-only OS environment where you can't execute any new code, it still won't stop you from owning yourself by falling for phishing and such attacks.
- Malware testing? I don't see it as being very useful except for marketing purposes and occasionally revealing a particularly clever rogue security software that is only revealed to be a fake by its utter lack of efficiency instead of any more obvious sign. It seldom tells Joe User anything that matters to them.
- AVs? The weakness of traditional anti-virus products is that they suck against new malware. Their strong side is that they require less interaction and brains from the user.
- HIPS/sandboxing/virtualization/LUA&Applocker etc? The strong side is performance against any malware or exploit regardless of age - no signatures required, so the restrictions they apply to malware apply to even unknown and new malware. The weak side? They more or less require the user to have some understanding of what he's doing and how the security measure works. Limited user accounts or sandboxes, for example, don't do much if the user just always gives anything and everything the admin password when asked and will execute anything outside the sandbox if the program bothers to pop up an error message when running in the sandbox. And there are loads of users who will do exactly this, unless you stop them either by taking away their admin password or threatening them with fire and brimstone.

[JamesFrance]

-{ Quote: "These are the same people who will click on a popup and bypass security anyway. :wacko:" }-

Yes, the people who need protecting the most!

So mainstream security needs to work for them automatically.

[Threedog]

-{ Quote: "Yes, the people who need protecting the most!

So mainstream security needs to work for them automatically." }-

And that's the problem. The best security solutions require the user to take a few minutes to learn a little bit. Automatic security that doesn't require a few minutes to learn doesn't always work out that well.

And in how long it would take to learn enough to save your bacon? How long does it take to read Blue's post about securing your computer? 15-20 minutes? But that little bit of education would go a long way to keeping you safe.

As for the malware test....the subject at hand....excellent results Defensewall!!!!!!!! :thumb:

[jmonge]

-{ Quote: "And that's the problem. The best security solutions require the user to take a few minutes to learn a little bit. Automatic security that doesn't require a few minutes to learn doesn't always work out that well.

And in how long it would take to learn enough to save your bacon? How long does it take to read Blue's post about securing your computer? 15-20 minutes? But that little bit of education would go a long way to keeping you safe.

As for the malware test....the subject at hand....excellent results Defensewall!!!!!!!! :thumb:" }-yes DefenSeWall the life saving bacon security program:thumb: :thumb:

[pjb024]

-{ Quote: "yes DefenSeWall the life saving bacon security program:thumb: :thumb:" }-

Did you enjoy your meal with your friend and, most importantly, did you manage to clean his PC? :)

[pjb024]

-{ Quote: "I'm going to have to side with Ilya on this one. Regardless of how good or bad this particular test is, real world experience shows every day that people are getting infected in spite of having security software like big-name commercial AVs or popular free AVs or "firewalls" installed and running on their systems. Who here has not seen an infected system that was running an up-to-date Norton suite or something? Fact is, against new malware, such as some trojan-of-the-day that was released 45 minutes ago on a drive-by-download site, most security software is of little use. AVs for example will not have the definitions for such a new malware yet, and if the AV happens to have some kind of HIPS or cloud component to protect against new malware, that still won't necessarily stop social engineering attacks where the user has been fooled to want to run the malware and will just click yes when warned that the file may be up to bad things.

As for default settings, sure, it's typically true that with some configuration security is improved. This is true for many things, first of all Windows, browsers, and even security software. The obvious problem is, how well can the average Joe User perform the configuration changes needed to increase security and how likely is he to even know about the possible need of changing the configuration to increase security? Not very well and not very likely is the answer usually. In real life, a vast mass of users are using everything with the default settings as far as security goes. We are dealing here with the kind of people who have not all yet discovered that there are other fonts than the MS Word default Times New Roman. These people rely on the default settings - if the default settings are not so good, then they are the ones who suffer from it. Compare, for example, how many Linux users run as root (root is not the default) with how many Windows users run as admin (admin is the default)...



I certainly believe security software testing in general is not worth much. Most tests are silly one way or another for reasons such as testing against a set of malware samples that is far too small to matter. Typically testing seems to serve only two purposes: 1) getting the testing group some name and 2) getting some folks switching from one security software to another in their endless search for the "best" and possibly shelling out a lot of cash to happy security product vendors in the process.

But I think the idea of only being able to get meaningful results by doing one's own tests is problematic. First, would the kind of users who actually need security software be qualified to test security software in any meaningful way? Can they build a safe testing environment, locate enough samples of malware and exploits, and then actually perform the test? Nah. Which leaves us with the thought that the only people qualified to do any meaningful security software testing wouldn't really need said security software to protect themselves and therefore might not have any reason to even want to test them...

Of course in new malware or exploit testing any kind of security feature not relying on signatures is likely going to be stronger than signature-based products. The problem with these non-signature-based methods is that they either directly require users to make wise decisions ("Unknown process foo.exe wants to perform some fancy technical stuff you don't understand and this may be dangerous, allow or deny?") which tends to be a decently reliable recipe for disaster, or at the least they provide ways for the user to still screw up magnificently unless he knows what he's doing ("Oh, look, this malware I just downloaded in my limited user account is saying that it needs admin rights, so I guess I'll just give the admin password now and see what happens" or "Hey, this program says it doesn't want to run sandboxed and untrusted. I guess I'll just run it in the real system then to get it working for realsies.") which is something that most users don't.

But before I digress further, where does all this leave us?
- The single most important thing in security is knowing what you're doing. If you don't, there's always the pretty real chance that you may get owned by incompetence in spite of having AVs and HIPS software, limited user accounts and what not. Even if you're in a read-only OS environment where you can't execute any new code, it still won't stop you from owning yourself by falling for phishing and such attacks.
- Malware testing? I don't see it as being very useful except for marketing purposes and occasionally revealing a particularly clever rogue security software that is only revealed to be a fake by its utter lack of efficiency instead of any more obvious sign. It seldom tells Joe User anything that matters to them.
- AVs? The weakness of traditional anti-virus products is that they suck against new malware. Their strong side is that they require less interaction and brains from the user.
- HIPS/sandboxing/virtualization/LUA&Applocker etc? The strong side is performance against any malware or exploit regardless of age - no signatures required, so the restrictions they apply to malware apply to even unknown and new malware. The weak side? They more or less require the user to have some understanding of what he's doing and how the security measure works. Limited user accounts or sandboxes, for example, don't do much if the user just always gives anything and everything the admin password when asked and will execute anything outside the sandbox if the program bothers to pop up an error message when running in the sandbox. And there are loads of users who will do exactly this, unless you stop them either by taking away their admin password or threatening them with fire and brimstone." }-

Nice post and I agree with everything you say. I was not seriously suggesting that users undertake tests of security products although I'm sure that some, many of who are posting regularly on Wilders, are more than capable of safely running their own trials.

My main gripe is that these so called tests rarely give out sufficient information on how testing was performed and people are expected to just blindly 'accept' the results as fact. If you are reasonably familiar with using a particular product and you sat in on these tests I imagine you would be critical of how the test was being conducted and you would maybe feel that the result is flawed due to the methodology being employed. Or maybe you would be impressed with the way the tests are being conducted ... but we have no way of knowing which scenario is nearest to the truth. Therefore, people should be cautious in using these tests as a measure of effectiveness.

I accept that DW, for example, would fare better against zero day malware attacks than a signature based scanner which will have little chance to prevent infection, although heuristics would maybe give some chance to prevent infection. However, I would not want to put all my faith in one product and I still believe that a multi layer defence is more effective than a single line of defence.

Of course, multi layer defence means more complexity and is this appropriate for the average user who may make his system more vulnerable to attack by misconfiguration? There is no easy answer because it depends on each user and his ability and desire to assimilate information and understand the ramifications of actions he takes. So does this mean that security products should be 'fool proof' and take away the complexity, or at least hide it, so that a good 'out of the box' experience will give maximum security for no investment in learning about how a particular product works? For some the answer is yes but for others, like most savvy Wilders posters, they would be unimpressed if the product hid all settings and did not allow any tweaking whatsoever.

The bottom line is that no product will ever satisfy all users and certainly not if it only works on standard settings and has no configurability. For any test to proclaim that product A is better than product B without giving a reasoned explanation of why that is so, and listing the caveats in reaching that conclusion, draws into question the validity of the testing methodology.

It's not good enough to just give a list of winners and losers and award gold, silver and bronze awards. That smacks of marketing hype. I use an AV which rarely gets included in tests and when it does it always gets low scores for detection and maybe I should be scared and change to another AV. I have tested many and keep coming back to the one I still use. I have never had an infection so that tells me that something may be working better than it should according to the tests. Recently I added a FW/HIPS and that gives me another layer. Previously I just relied on my router FW. I still didn't get infected so maybe my AV, which gets poor results in the tests, works better than the tests give it credit for.

If I took the tests as being the definitive means to choose an AV then I would be using a different product. The only problem I have is not speaking Russian and most of the forum activity for my AV is in Russian. ;)

I just urge people not to blindly accept that these tests are meaningful and don't use them to choose your protection. The best way is to try different products until one 'feels' right then stick with it unless there is an overriding reason to change. If it ain't broke don't fix it!

Finally, I'm not trying to spread a doctrine. I do what I do and you are free to take whatever approach works well for you. And, whatever you do, be safe!

[Kees1958]

Windchild,

Nice post, also a self declared nice profile you have got shameless LUA troll ;D

In regard to LUA+SRP
By the way I am running LUA + SRP on XP Pro with only Avast file shield and windows FW, because a LUA setup reduces the attack surface with 80% at least. My Son is a fanatic gamer he is using UAC + Sully's PGS + Vista FW-2 way and uses on demand AV only. For him the need for speed and big game updates (very time consuming to re-install after a serious virus infection), is a reason to accept the (user friendliness) limitations of the OS its security features.

In regard to DW
I tried LUA with SRP on my wife's PC, but the fact that she is not allowed to run a new program on HER pc by some stupid security program or click on time (XP box) to check when she can make an appointment with friends is just to limiting. On top of that she is a click OKAY/YES happy pc user.

Normal PC users (mind you not hobbyists), use a PC for its function (webbrowsing, e-mail, digital media files, office aps), she rarely installs programs.

DW makes it possible to enjoy a stronger than LUA seamless protection (no in or out of the sandbox hassle). DW always was very quiet, with the new whitelisting feature (to facilitate safe installs) of the V3 HIPS/FW it is near silent, at least silent enough to keep a very non-tech and critical PC user happy.

Why don't you try DW, not for yourself, but to understand how close this is to (stronger than) LUA + SRP protection and how its user frienliness is close to the ease of use of an AV. From this point of view it is understandable for me why DW was included in this test (it is as easy to use as an AV)

Regards Kees

[Windchild]

-{ Quote: "
Why don't you try DW, not for yourself, but to understand how close this is to (stronger than) LUA + SRP protection and how its user frienliness is close to the ease of use of an AV. From this point of view it is understandable for me why DW was included in this test (it is as easy to use as an AV)
" }-

With regard to the wife's PC, you could simply open up SRP rules a little to allow her to execute new apps from some folders - and this would still prevent the average drive-by download attack. With Win 7 and AppLocker's publisher rules, that kind of stuff is easier still than before.

As for DW, I think I understand well enough the kind of protection, benefits and downsides it has. I certainly see no reason why one could not test DW together with AVs against malware of various kinds, but then one should expect the signature-based products to lose such tests. It's not all sunshine and honey, though, for products such as DW. Like with LUA (unless the user does not know the admin password), there's always the chance that the user will just tell DW to trust some random file they really should not trust, instead of letting it remain untrusted. Hard to do anything against that - but the good side to AVs is that they may just automatically eradicate a known bad file without even giving the user any option to trust it. As far as my personal preferences go, DW and HIPS products in general really aren't for me or something that I feel comfortable recommending to average users. One reason obviously is money: many users are unwilling to shell out the bucks for a security software - or any software, having spent a lot of money on the computer hardware already. But for me, the really big issues are like this:

-{ Quote: "FireFox slowdown because of DW is about two or three seconds at my computer. Well, four maximum." }-

That is to say, issues like possible slowdown and compatibility and stability problems, minor or major. These things are of course not DefenseWall issues so much as issues with security software in general, which is one huge reason why I prefer my brains and security built into the OS over other options. Right now, in a non-English speaking forum I'm helping a guy who has his AV's driver causing blue screens after yesterday's Windows updates. I've seen countless such cases where security software BSODs on systems with perfectly fine hardware and no malware infections. I choose not to trouble my own systems with that stuff. I prefer fast and stable. And it just so happens that on my systems, slowdowns and crashes are like dogs that speak Norwegian - very rare! ;)

But sure - if some cosmic force made me choose between either using only DefenseWall / some other HIPS type product for protection or only an AV of my choice, I would certainly choose DW/HIPS over the AV, because the AV just doesn't perform as well against malware attacks assuming a user such as myself who isn't completely clueless on how one is supposed to use the security software.

That's really one reason why I find tests like the one this thread is about so... well, pointless. "HIPS beats AV against new malware!" the testers proclaim. That's like saying: "In other news, rain is wet, so use an umbrella." ;D

[Ilya Rabinovich]

-{ Quote: "In other news, rain is wet, so use an umbrella." }-
Unfortunately, majority of the computer's users have no idea that rain is wet. Other words, they are aware of AV's only.

[Scoobs72]

-{ Quote: "With regard to the wife's PC, you could simply open up SRP rules a little to allow her to execute new apps from some folders - and this would still prevent the average drive-by download attack. " }-

Why would I go with a solution like that that only stops the 'average' drive-by when DW gives much stronger protection with far less hassle? DW is on my wife's PC and I haven't had to touch it in months - it requires zero management from me.

[hawki]

How can you have any confidence in this group? They are testing for zero-day attack effectiveness and for Norton they use NIS 2009 when NIS 2010 has been out for several months with its new Insight protection that has been shown to be awesome in blocking zero-day stuff.

You have to wonder.

[Kees1958]

-{ Quote: "With regard to the wife's PC, you could simply open up SRP rules a little to allow her to execute new apps from some folders - and this would still prevent the average drive-by download attack.
" }-

Well I tried that, just it is not as simple as you say. Some installers unpack into temp others into the parent directory, leaving open the two nastiest hole I can imagine, Temp folders and User space main directory, which practically reduces SRP to zero. My bet is that DW is the most used security application for other family members (like Scoobs, JJMonge, and others).

I think DW is a lot cheaper than a Windows 7 upgrade, on XP Chrome starts within a sec cold, subsequent startups are in a blink of an eye.


-{ Quote: "
With Win 7 and AppLocker's publisher rules, that kind of stuff is easier still than before.
" }-

I do not need Applocker's publisher rules. Try PGS, a really brilliant freebie to implement SRP on application name and even wildcards in the name like wind*.exe or *child*.* , ten times easier than Applocker's publisher rule's.

[Windchild]

-{ Quote: "Unfortunately, majority of the computer's users have no idea that rain is wet. Other words, they are aware of AV's only." }-

That is very true. But it's not easy to change this state of affairs, unfortunately. The majority of users pay little attention to anti-malware tests if they even know about them at all. And I really don't think that's a bad thing, considering the quality of the tests in general. Better to educate the users by other means that are more understandable and clear, and less... well, financially motivated.

-{ Quote: "Why would I go with a solution like that that only stops the 'average' drive-by when DW gives much stronger protection with far less hassle? DW is on my wife's PC and I haven't had to touch it in months - it requires zero management from me." }-

You wouldn't, if you don't think it fits your preferences and needs. I was simply saying what could be done in a case where someone was using SRP and wanted to be able to execute some new programs. That someone might have chosen SRP because of the price or lack thereof, for example.

-{ Quote: "Well I tried that, just it is not as simple as you say. Some installers unpack into temp others into the parent directory, leaving open the two nastiest hole I can imagine, Temp folders and User space main directory, which practically reduces SRP to zero. My bet is that DW is the most used security application for other family members (like Scoobs, JJMonge, and others)." }-

Sure, some installers will have trouble. But then, if this is a limited user account, that would be the case without SRP, as well, seeing how most installers don't like LUA. One could always install legit software as admin and continue along happily. Where this is not possible or is too much of a hassle, then there are fortunately other options, like the DW that you're using.

-{ Quote: "I think DW is a lot cheaper than a Windows 7 upgrade, on XP Chrome starts within a sec cold, subsequent startups are in a blink of an eye. " }-

Certainly DW is cheaper than a Windows 7 upgrade, but I wasn't telling anyone to upgrade to Windows 7. Instead, the point was that for those who use Windows 7 there are nice improvements. As for Chrome, not exactly my favourite browser. And even if it was, there surely are other programs one might wish to run like Firefox, some of which will suffer from noticeable slowdown at startup, and other things.

-{ Quote: "I do not need Applocker's publisher rules. Try PGS, a really brilliant freebie to implement SRP on application name and even wildcards in the name like wind*.exe or *child*.* , ten times easier than Applocker's publisher rule's." }-

There are two issues here:
1) You can make filename based rules with wildcards without PGS just as easy as you can make other path rules. Unless you're using a Home version of Windows, but I wouldn't use those. The XP Home version for example was notoriously dumbed down to the point of being an absolute pain to use for me.
2) Filename based rules are also about ten times less secure than publisher rules. You can rename ikillyourpc.exe to wind.exe and now it will run, if you use filename rules. But if you use publisher rules that allow only files signed by certain trusted parties, that would not work, because the file still will not have a valid digital signature from one of your trusted parties. So, while AppLocker's publisher rules aren't an essential feature, they can come in very handy in some cases.

But, I digress. To return to the subject of malware testing, I've always found it amusing that a simple LUA setup is almost never included in such tests (and a LUA with SRP setup is never included). That is one more reason why security software tests in general are not worth anything much to me - they don't even bother to make a rough comparison between the security software and the operating system's own security measures. I continue to be amazed by how much some people care about such tests in spite of the fact that the tests are massively unreliable at worst and captain obvious at best.

[Kees1958]

Windchild

Security should be user friendly to the one using the PC, for me policy and access management are the way to go (yes I am deformed by old days of VMS/RACF etc), therefore all users at home have a different setup

a) Every day - average user: XP Home = admin with DefenseWall V3

b) Gamer - media junkie: Vistax64 = UAC + Norton UAC + SRP (through PGS)

c) Elderly IT geek: XP Pro = LUA + SRP and ACL

So from a troll's view of point, we nearly share the same insights, with the difference that my scope is a bit broader than LUA, being policy management, see new signature. ;D

[JerryM]

Interesting, and I was just thinking. ??? ???

Regards,
Jerry

[andyman35]

-{ Quote: "How can you have any confidence in this group? They are testing for zero-day attack effectiveness and for Norton they use NIS 2009 when NIS 2010 has been out for several months with its new Insight protection that has been shown to be awesome in blocking zero-day stuff.

You have to wonder." }-
Yes I made that point myself,the test is certainly skewed in favour of the products tested with the latest versions.

[JamesFrance]

It does say that it is a long term test started in July, so they used what was available then.

[Fly]

About DefenseWall:

I have never used it.
Does it offer complete protection against spyware/adware/phishing etc. ?
A hypothetical example: you visit an infected website (as intended by the owner, or a hacked website) that contains a malicious script, and that script, possibly with a trojan or rootkit, captures your credit card data or other information. Does DefenseWall protect you against that ?
Generally, an AV would. The AV would detect the malicious script, the payload etc. The AV may have some http:/ or link scanner. The McAfee SiteAdvisor would be a bad example of protection against malicious websites. :)

Next subject: IMO, the matter of security (viruses, spyware) is primarily the responsibility of the user/admin. The user/admin should know what is safe and what is not, and how to deal with uncertainty. No security software will fully protect you against bad decisions. IMO, security software should complement the decisions of the user/admin.
It's safer to 'surf' the web when you know what you are doing without using security software (with the exception of a firewall), than not knowing what you are doing and just relying on security software.

It's really not that hard to learn, and on the long run one saves time, effort and cost if the user/admin learns about IT/internet security.
Unfortunately, few users learn. Any ideas on how to change that ?
Some trust AVs, some choose a HIPS, some a 'tech' solution like LUA+SRP.
Someone stated that LUA+SRP reduce that 'attack surface' by at least 80%.
That still leaves 20 % ...

IMO, it's all about the mind of the user/admin.

I use an AV, but I could probably do without it since I haven't gotten infected for over at least a year.

[Ilya Rabinovich]

-{ Quote: "Does it offer complete protection against spyware/adware/phishing etc. ?" }-
Spyware/Adware/Keyloggers/Rootkits/Trojans- yes, phishing- no, it's your browser's job.

-{ Quote: "
A hypothetical example: you visit an infected website (as intended by the owner, or a hacked website) that contains a malicious script, and that script, possibly with a trojan or rootkit, captures your credit card data or other information. Does DefenseWall protect you against that ?" }-
A script can not be with a trojan or rootkit. It may use some techniques to drop maliciuous executable modules (that may contains trojan, rootkit, you name it) at your computer and execute them.

-{ Quote: "
Does DefenseWall protect you against that?" }-
Yes, sure. That's the main point of the project.

-{ Quote: "
Unfortunately, few users learn. Any ideas on how to change that ?" }-
Nobody can change it. It's "by design". :)