Independant antispyware testing?

Is there an organization or company that does objective, thorough, 3rd party testing and reviews of antispyware/amtimalware products? You read reviews in PC Magaizine, CNET, or other sites, but how do you know if they have been thorough and objective? Does standard protocol exist for antispyware testing?

For example, you might read a review that compares 5 or 6 antispyware products. But then on careful review find that they only tested with 20 or so malware samples. How accurate can that test be with such a small sample of malware? And were those 20 samples randomly picked? Selectively picked to make a certain product look better?

How do you know which sources of information to trust and which products are the best?

 

This is only my opinion, but I rely on the members of this forum for my information. Never had a problem (thank you, Wilders Folks ).

 

Eric Howes over at Spyware Warrior did some comprehensive testing on anti-spyware programs, granted they are somewhat dated, but perhaps still worth a look. Last I heard from him he is planning on doing another comprehensive test in the near future when he gets more time. Below are some links to his anti-spyware test and comparison:

http://spywarewarrior.com/asw-test-guide.htm

http://spywarewarrior.com/asw-features.htm

This test is the only one I could think of off-hand. There are many others, some less reliable and objective than others, but look at them all critically, don't just assume they're accurate and honest.

 

Thorough testing of multiple anti-spyware apps is practically impossible. The best that can really be done is to test them against a sampling of the current malware and report on how they did against those specific samples. There's far too much malware in circulation to be thorough. Even if it were possible to collect all of the malware in circulation, it would take so long to test with all of it that the results would be outdated before the testing was done. If you include the different variants, there's thousands upon thousands of samples, with more being released daily.

One of the biggest problem with anti-spyware tests is choosing the test material. There's no such thing as a typical or random infection. Different scanners have different strengths. One may do better against conventional trojans. Another may be better against the well known malware variants. The tester can make the test results say anything they want them to by manipulating their choices of material. Unfortunately, a lot of "testers" aren't truly independent. Unless the tests are sponsored by an anti-spyware vendor, in which case the partiality is obvious, the reader has no way of knowing just how impartial the test procedures are. The people most suited to do independent testing, those who are on the front lines of the fight, aren't inclined to do comprehensive testing, primarily because of the futility of it. The results are valid for an extremely short time, less time than the testing takes. Malware writers update and release new variants incredibly fast, often for the sole purpose of evading detection. Some change almost daily.

Eric Howes tests were pretty well done. Although the malware sampling tested was small, it was sufficient to show performance patterns in the products at the time. Some had false positive problems. Some were consistently missing most of the samples. Even with the small test sampling, the test results were outdated very shortly after he published them. The better anti-spyware apps have been updated since then and the malware itself has been modified repeatedly.

The best you can expect from anti-spyware apps is that they will detect a varying percentage of the malicious code in circulation. They can remove much of what they detect, but not all of it. The specific items and the percentage of malware detected are in a constant state of change. None of them catch everything. None of them can remove everything they detect. None of them are completely free of false positives. An independent, unbiased test like Eric's bears this out. By comparing several independent tests over a period of time, it may be possible to see patterns and trends with the specific products and their vendors. One might have a high detection percentage and a large number of false positives (heavy reliance on heuristics). Another might always miss adware/malware from a specific company (possible affiliation). Yet another might always have low detection rates or can't remove what it detects (possible rogue product). That's about as far as the testing that's being done can go.
Rick

 

Perfect summation herbalist.

Hence HIPS are a very popular/neccessary and effective countermeasure if not compliment to scanners. Simply put.

 

Yes because we can't test with 100% certainty , we shouldn't test at all.

If you criteria of using a solution is whether it can be tested reliably I think you should abandon HIPS, since the methodology for testing HIPS is even more uncertain!

 

Everyone's machine is different. From manufacturer to O/S.

HIPS are my personal DEFENSE WALL of choice and namely System Safety Monitor which is performed well beyond my wildest expectations.
It's of course based on rules you set via pop-up alert boxes to sudden interactions that have started.

You can take anti-malware apps reviews on face value if you like, but i don't. I run local research and most aggresively i might add including rootkits, some of which are not easily available thru normal surfing of URL's.
Even so, there are also DEMOS galore if your so inclined.

HIPS testing "methodology", although not completely infallible is quite sufficient enough to a point. Open discussion bears this out.

Herbalist rightly offers explaination to the various strengths & material used as to do regarding comparisons offered as suitable to our confidence. I take results like those published in public with a grain of salt, local research on your own table, for me anyway, removes any grey areas of concern published. For me it's the only way and reliable since i never am forced intruded upon. Just isn't happening.

 

Btw, a bit OT, but I never really understood why people have so much faith in signature based products from the smaller companies (A Squared, Spyware Teminator, BO Clean to name a few).

My personal opinion: the smaller the company the less likely that they can keep up with all the malware that´s released everyday. I mean you need a lot of people to come up with new signatures everyday, not? That´s why I would never buy any AV/AS/AT product from a small company. My question is, am I right or am I wrong?

 

Wrong actually!

Ewido was a small company and a lot of us were happy to buy the product based on performance.

A lot of folks round here salivate at the thought of SAS, whose sigs seem capable of cleaning quite a few things the big boys can't.

As for A2, the most important part of it (the Guard's IDS) is not even sig based.

 

The best service is generally provided by smaller companies, and that holds true in most industries not just security software. So when your favorite small company is bought up by a corporate giant, the service usually goes to crap.

 

Kudos to Herbalist for summing it up quite well
All AS apps (and all software and hardware "security" for that matter) are a pure defense scenario - always trying to play catch-up; by the time they detect one of the nastier malware or variants, chances are the damage has already been done and the bad guys have what they were looking for.
Layered security is the best anyone can hope for, short of not being connected to the Net. It will always be a defensive situation for the end user.

 

No doubt.

Funny you should mention "defensive situation" that we're destined to live with, i always envisioned at some point down the line over the years that a method might one day arise where we could return any forced intrusion directly back to it's source in some endless loop that would jam the transmission of that crapola and overwhelm them at the source as reward for their effort.