Webroot SecureAnywhere 8.0.0.7 AV on-demand testing

[lordraiden]

After reading some people commenting the low performance in detection of Webroot SecureAnywhere have decide to test it myself.

I have tested a total of 3038 (0day and recent malware) files and the results are.
Webroot SecureAnywhere 1599 52,6%
Emsisoft Free 2807 92,4%

Also Webroot SecureAnywhere has failed to delete any of the files (must be a bug)

There is any reason for this low performance? it's related with the beta?
Even if the cloud is not 100% ready we should expect a better protection, if not I don't even want to imagine what would happens with the computer offline.

[andyman35]

According to PrevX Help in another thread,when asked about it's current strength:

Originally Posted by PrevxHelp

We'll be having several organizations test it as soon as it is ready. It is still beta software so it isn't as strong as it will be but you can always use it with another AV in the meantime if wanted. If it means anything, I use only WSA

I'd expect it to be much better once it's final and of course it's traditionally been a lot stronger real-time than on-demand.

[lordraiden]

Originally Posted by andyman35

According to PrevX Help in another thread,when asked about it's current strength:



I'd expect it to be much better once it's final and of course it's traditionally been a lot stronger real-time than on-demand.

Ok, I will check again with the final version.
Maybe it's time to include Webroot in AVC

[Rivalen]

Originally Posted by lordraiden

After reading some people commenting the low performance in detection of Webroot SecureAnywhere have decide to test it myself.

I have tested a total of 3038 (0day and recent malware) files and the results are.
Webroot SecureAnywhere 1599 52,6%
Emsisoft Free 2807 92,4%

Also Webroot SecureAnywhere has failed to delete any of the files (must be a bug)

There is any reason for this low performance? it's related with the beta?
Even if the cloud is not 100% ready we should expect a better protection, if not I don't even want to imagine what would happens with the computer offline.

In my thread Joe said it had all the protection of Prevx + all new stuff in WSA. Was Prevx ever tested anywhere? Confusing - 52% is so poor if Prevx is the base.

Best Regards

[PrevxHelp]

Originally Posted by lordraiden

After reading some people commenting the low performance in detection of Webroot SecureAnywhere have decide to test it myself.

I have tested a total of 3038 (0day and recent malware) files and the results are.
Webroot SecureAnywhere 1599 52,6%
Emsisoft Free 2807 92,4%

Also Webroot SecureAnywhere has failed to delete any of the files (must be a bug)

There is any reason for this low performance? it's related with the beta?
Even if the cloud is not 100% ready we should expect a better protection, if not I don't even want to imagine what would happens with the computer offline.

Could you please send me a log after running the scan over the folder to report@prevxresearch.com so that I can take a look? That is definitely far lower than what we would expect and it's far lower than private testing being done by 3rd party testers has shown as well.

Thank you!

[shadek]

My tests show something different... this is what I wrote a few days ago;

"I used to be able to copy ~12.000 fresh malware per day and around 2.000 would remain after WSA did its job. Now a lot more samples are detected by Malware.Generic definitions...leaving around 500 samples left for each batch... is this a co-incident or is it great engineering?"

Note that most days after that has shown the same result meaning the detection rate for on-demand is around 96%, and a lot of my samples are 0-day. So my tests show something different, I'm not sure why. If I try to run the remaining 4% of the samples, WCA usually detect the file as malicious within seconds after execution via the suspicious behavior. Only a few, most of them rouge software, are passing through protection and are left running wild in OS.

[Triple Helix]

Originally Posted by shadek

Only a few, most of them rouge software, are passing through protection and are left running wild in OS.

@ PrevxHelp In this case would going in the System Tools Tab>System Control>Control Active Processes can you kill the Rogue's processes?

TH

[shadek]

Originally Posted by Triple Helix

@ PrevxHelp In this case would going in the System Tools Tab>System Control>Control Active Processes can you kill the Rogue's processes?

TH

Attachment 228905

Yes, I can easily kill them! No worries there. I'm just talking about the detection rates/prevention rates! The intervention rate is probably close to 100% with WCA.

EDIT: By the way, killing the processes doesn't mean all the files/registry keys the rouge software installed are removed completely!

[Triple Helix]

Originally Posted by shadek

Yes, I can easily kill them! No worries there. I'm just talking about the detection rates/prevention rates! The intervention rate is probably close to 100% with WCA.

EDIT: By the way, killing the processes doesn't mean all the files/registry keys the rouge software installed are removed completely!

No I understand as most AV's have trouble with Rogeware and this is where I would like to see WSA improve upon as then we don't have to rely on other scanners to clean up the leftovers!

TH

[PrevxHelp]

Originally Posted by shadek

Yes, I can easily kill them! No worries there. I'm just talking about the detection rates/prevention rates! The intervention rate is probably close to 100% with WCA.

EDIT: By the way, killing the processes doesn't mean all the files/registry keys the rouge software installed are removed completely!

Adding the file with Manual Threat Cleanup should remove any registry keys/files created as well if WSA was installed before the infection

[Triple Helix]

Originally Posted by PrevxHelp

Adding the file with Manual Threat Cleanup should remove any registry keys/files created as well if WSA was installed before the infection

WoW great to know! Your going to have a big list to give us for us to know all the capabilities of WSA!

TH

[shadek]

Originally Posted by PrevxHelp

Adding the file with Manual Threat Cleanup should remove any registry keys/files created as well if WSA was installed before the infection

Thanks for sharing the information, Joe. Based on this new knowledge, I have something on my mind. I'm thinking of doing an extensive on-execution test and share the numbers with devs at Webroot (aka PrevxHelp) and then after that to all here at Wilders. Now that I know how to clean-up missed detections properly it's going to be a lot of fun!

The test will consist of 0-day malware, rouges, rootkits, etc. I will include detection rates, prevention rates, clean-up rates. MD5s will be provided to the public users here at Wilders while missed samples and all tested samples with MD5 will be provided to the staff behind Webroot. Around 100 samples will suffice for an on-execution test, don't you guys think?

P.S. We're not talking samples off MDL or malc0de. I have an entirely different malware pool.

[PrevxHelp]

Originally Posted by shadek

Thanks for sharing the information, Joe. Based on this new knowledge, I have something on my mind. I'm thinking of doing an extensive on-execution test and share the numbers with devs at Webroot (aka PrevxHelp) and then after that to all here at Wilders. Now that I know how to clean-up missed detections properly it's going to be a lot of fun!

The test will consist of 0-day malware, rouges, rootkits, etc. I will include detection rates, prevention rates, clean-up rates. MD5s will be provided to the public users here at Wilders while missed samples and all tested samples with MD5 will be provided to the staff behind Webroot. Around 100 samples will suffice for an on-execution test, don't you guys think?

P.S. We're not talking samples off MDL or malc0de. I have an entirely different malware pool.

That sounds fantastic It's very similar to what we do internally on a day-to-day basis so it will be great to see what your samples return to get a picture of the malware you're seeing.

Let me know your results or if you want anything different from my end to help

[lordraiden]

Originally Posted by PrevxHelp

Could you please send me a log after running the scan over the folder to report@prevxresearch.com so that I can take a look? That is definitely far lower than what we would expect and it's far lower than private testing being done by 3rd party testers has shown as well.

Thank you!

I have sent you the log, if you need something else let me know.

About the testing I must to say that I was testing a full package with script, dlls.... testing just the exe files the result is
Scan Results: Files Scanned: 2039, Duration: 1m 0s, Malicious Files: 2019

So pretty good

Anyway Emsisoft was able to detect as malicious most of the dlls, scrips, binary files...

Tomorrow I will repeat it again with new exe files.

[PrevxHelp]

Originally Posted by lordraiden

I have sent you the log, if you need something else let me know.

About the testing I must to say that I was testing a full package with script, dlls.... testing just the exe files the result is
Scan Results: Files Scanned: 2039, Duration: 1m 0s, Malicious Files: 2019

So pretty good

Anyway Emsisoft was able to detect as malicious most of the dlls, scrips, binary files...

Tomorrow I will repeat it again with new exe files.

Thanks for the logs - we received them here. I haven't heard of that website but it certainly is interesting The on-demand/right click scanner only uses a small local database against scripts/non-executable files because of the possible privacy issues associated with sending documents/PDFs/etc. to the cloud. If a file actually tried to threaten the PC from a script, it would be blocked but we don't try to scan these when out of context.

DLLs are handled like exes, however, so you should see good detection for those as well.

Thanks for the testing!

(A note regarding the "removal not completed" - there are a few cases where the engine will detect if the system is so bad off that it needs a support engineer to assist. When finding 2000+ infections, I'd think the user would be in pretty bad trouble This might need to be changed to handle people intentionally doing on-demand tests like this but for the average user, we're trying to make the process as easy as possible for them without potentially breaking applications on their PC)

[shadek]

Originally Posted by PrevxHelp

That sounds fantastic It's very similar to what we do internally on a day-to-day basis so it will be great to see what your samples return to get a picture of the malware you're seeing.

Let me know your results or if you want anything different from my end to help

I am pleased that you're delighted with this test and that you support it. I will begin gather varied samples tomorrow! I will also describe the methodology and inclusion/exclusion criteria! So if there are any doubts, one can know for sure how the test was conducted and people will be able to criticize the methods.

[Triple Helix]

Just a heads up! Lets not get into this versus that or the Wilders staff will close the thread as it's not aloud in the AV section! But it will be nice to see a few results!

TH

[shadek]

Originally Posted by Triple Helix

Just a heads up! Lets not get into this versus that or the Wilders staff will close the thread as it's not aloud in the AV section! But it will be nice to see a few results!

TH

Of course. The test I'm going to conduct is just to evaluate a beta-product's performance!

[lordraiden]

Originally Posted by shadek

Of course. The test I'm going to conduct is just to evaluate a beta-product's performance!

But it would be nice to have a point of reference like Norton or Fsecure

[shadek]

Originally Posted by lordraiden

But it would be nice to have a point of reference like Norton or Fsecure

I will not do that. Firstly, I am not really interested in testing other products. Secondly, the test is there to test WCA beta's detection/prevention/clean-up capabilities.

[lordraiden]

Originally Posted by shadek

I will not do that. Firstly, I am not really interested in testing other products. Secondly, the test is there to test WCA beta's detection/prevention/clean-up capabilities.

Yes, but if it scores a 60% or 89% or a 95% will be a good result, a bad, normal? what is the point of the test if you can't qualify the result?

[andyman35]

While comparitive results can be informative,it's my understanding they won't be allowed on Wilders due to A v B unless from a professional organisation.

[lordraiden]

Originally Posted by andyman35

While comparitive results can be informative,it's my understanding they won't be allowed on Wilders due to A v B unless from a professional organisation.

It's not allowed to discuss about which one is better, something totally different from posting the results of 2 AV's or even better, 3.

[kero68]

"Note that most days after that has shown the same result meaning the detection rate for on-demand is around 96%, and a lot of my samples are 0-day. So my tests show something different, I'm not sure why. If I try to run the remaining 4% of the samples, WCA usually detect the file as malicious within seconds after execution via the suspicious behavior. Only a few, most of them rouge software, are passing through protection and are left running wild in OS."
Sadek, what heurestic settings have you configured to get good results like this?

[pykko]

I've just made a small test and review.

Regarding the on-demand scanner is excelent: I've used 7722 malware samples from July and August 2011.
Detection rate was: 95.76%, a very good one

For additional details, see the full review: http://www.faravirusi.com/2011/09/08...tie-excelenta/