Matousec & Webroot ?

Comments ?

( From Dec 2012 )
Webroot tests: http://www.matousec.com/projects/pr...oot SecureAnywhere Complete 2013 8.0.2.66.pdf

All products:
http://www.matousec.com/projects/proactive-security-challenge-64/results.php#products-ratings

 

WSA intentionally avoids leaktests like this - I'm surprised it even scored 11%. This has zero bearing on the actual protection that any of the tested products provide.

 

1: AV Comparitives has been the first and not the last to admit that their testing methodology doesn't even begin to accurately portray the functionality of WSA. Perhaps this tester needs to figure out what they are testing before they test it.

2: What is the methodology? Details! Any good testing organization should list at least the basic methodology information of their testing along with the results so that the results cannot be taken out of context or easily dismissed. This tester has failed at that very basic premise, which tends to cast doubt and cause their test to be dismissed. What does "Autorun31" mean? Absolutely nothing to anybody looking at the results. Why should any educated user (which are the ones generally using and telling other people to use WSA) take this with more than a grain of salt or bother to go look for the details if they are not given up front like a reputable organization would?

3: The testing methodology, JUST from the presented (very limited) data is flawed already. The test makes it clear that "If the package doesn't reach a certain score on a level, we do not continue testing, but then consider them failed for the further levels anyway." Auto-failure of 90 tests and those "failed tests" included in the "score" because of a "level system"? What do they think this is, some sort of video game?

In summary: I see no reason to take this tester seriously. Far too flawed a methodology apparent from the limited data that shows further flawed reporting methodology.

 

And with bit defender and norton among some of the big players also scoring poorly, i think these results should be taken with a pinch of salt

 

Matousec's tests test for HIPS-like features, so only products with an extensive HIPS will score high.

 

I'm not going to labour the point here, but:

From Webroot community 2012:
http://community.webroot.com/t5/Web...virus/Scanning-PC-suspiciously-fast/td-p/6632


So wrt to PrevX Help response: effectively, there is no current metrics for testing WSA ?.
I kind of know we have been down this pass before.
I am cognizant of the discussion with AVC and understand the complexity of testing cloud based tools HIPS, monitors and rollback tools. STill, some of the tests as reported are startling. It's not so complex when other vendors simply detect the the files as presented as malware.
There was an apparently uncontested message from Webroot admitting to not having the latest db for malware in there somewhere.
Never seen a good explanation of thr FPs.

One problem I've never seen addressed is the issue with zeroday malware: WSA allowing installation and "theft" while it (WSA) works out the "workings" of the mal. This is a point that does crop up in many discussions and never eally addressed.

Essentially a zeroday can install and then do its stuff, post back to home base and then WSA can roll back as/when the mal is detected, and if required WSA can sort out removing the mal remotely: might be a bit late if some goodies have been hoovered out? Happy to be corrected.

I have been reading about "Unique approach"I am happy to be educated about any and all aspects of this
http://community.webroot.com/t5/Sec...mparatives-and-Our-Unique-Approach/td-p/12380

I am happy to have a bunch of links fed to me rather than Joe having to go round the garden again.

OK
Cant comment on BitDefender: no experience whatsoever.
Symantec, as industry leader not always loved or highly regarded.

It cant really be enough for Webroot to have a number of commited users saying : "I never have an infection", there must be something more ?

 

This has been discussed several times before. First local heuristics depending on the action of the malware will act before then later this is especially true if the malware start to manipulate system or sensitive PC areas. Secondly the other modules of WSA (e.g. indentity protection) will prevent user information to leak out or jamm the information leaked rendering it useless (even if the malware is not yet identified as such).

 

Not every HIPS is the same. Matousec tests products which show a prompt on every suspicious action - something we have no interest of doing. We analyze the changes which are made and bring them up to our cloud to decide, but we've intentionally whitelisted all of their leaktesting tools because they complained when we blocked their files... to pass these tests, you have to allow the files to run but prompt on their actions, so because WSA is able to make a decision about files when they aren't actually making malicious changes or stealing data (and therefore allowing the leaktests to run), and blocking files isn't allowed, this isn't at all like a normal world test and we're not going to devote resources just to score well in it without improving the real world.

We're actively working with AV testing companies to try to get WSA tested correctly. Here are a few thoughts on AV testing right now with WSA: http://blog.webroot.com/2012/07/19/webroot-bulletin-regarding-av-comparatives-results/ (and see the comments as well for a bit more Q&A dialog)

 

Good news! You can be happy. ^.^

"Do its stuff" for a 0D involves either grabbing information or forming a stealth beachhead. It's going to be an unknown, so it will already be monitored, which also means blocked from certain system access. Things that a 0D would normally grab will result in null datasets on its endpoint and the cloud comparison of multiple copies of the zero day means that suddenly, instead of one program going (as a lame, but not giving away all the secrets example): "Screenshot (comes up blank), then contact this server in Russia", there will be dozens or more. When one does it, it could be legit, so the normal HIPS is left unsure, but seeing 10 or 100 computers suddenly do that flags in the cloud so suddenly either the programmatic decision can be made or a human being can look into it.

If it tries to form a beachhead, it still needs to grab a payload and try to dig in. A smart threat will not try to dig in immediately, instead dropping a NON-Suspicious autorun of some sort that the user would expect and allow in conventional HIPS, effectively whitelisting the future action hidden elsewhere in the code. The cloud monitoring means that the future action is seen as well and it just went from the nice guy down the street to the bad guy you never expected. If it grabs a known-bad payload or a payload that becomes known as bad, both it and the payload get wiped out.

Though, please do feel free to let me know if you have any other things you think a Zero Day may try to do and I will see if I know how WSA would prevent that, or it may be something that could be improved. Just don't get too out there, otherwise the line of inquiry just gets annoying. And it's nice to keep in mind: if somebody finds some way to get past it temporarily (there always will be, no matter what, because criminals are smart and users are users), the development team blocks the hole within a VERY short time.