Is antivirus testing corrupt?

"My computer hasn't had a malware infection in YEARS and according to the latest "review" in a major magazine, my product was in the bottom half. Hmmm, kind of makes you wonder, doesn't it?"

My computer hasn't had a malware infection - as far as I'm aware, and I don't run any on-line security. Hmmm, kind of makes you wonder, doesn't

 

No, why should it? Results such as these follow some type of statistical distribution. They have wings well off the mean on both sides with the zero side bounded by..., ummm, zero.

I wonder why everyone seems to expect that everyone else should experience the same challenges and results that they do...

Blue

 

Results like these purport to show how successful or otherwise different programs are when presented with a number of malwares. They do not really address the question as to what is the probability for any user of being attacked. Do I really need to know that watch "A" is waterproof to a depth of
100 meters whilst "B" is only good to 90 if I only occasionally get caught in the rain ?

 

No, you don't, and that's my point. Staying with the analogy, the folks who dive to 100-120 meters may be interested, or not...., it depends.

Blue

 

Off topic: Most developers do not actually need to know anything about the PE file structure. A rough understanding of function import/export is sufficient. That's perfectly OK. There are plenty of skills that are lacking more than that in the developer community (e.g. notions of policy-based programming).

 

I don't buy your reasoning.

Its all statistics. I can test a new drug without knowing the biochemical interactions for the drugs. You get your population and simply measure the remission rate. Do it for different samples and then using statistics get the confidence measure. When possible, it’s the preferred way to test. This way you don’t introduce a “medical” bias.

You also don’t have to disassemble the virus samples. If 16 out of 20 anti virus makers say a sample is a virus then assume it is so. Maybe with this sample the other 4 were correct, but with thousands of samples there would be very few exceptions of this type and therefore the error would be statistically insignificant.

Similar to flipping a coin. You flip a few times you can get heads 30 to 70%. You flip thousands of times, then its rare that heads will vary from 50%. The law of large numbers applies.

So, if someone like IBK of A-V comparatives says an anti virus product is 84% effective, assume it is very close to that.

 

Actually, this is a flawed analogy.

If you go with this picture, replace a fraction of the active dosage forms with placebos. This isn't the same as a blind or double blind study pairing active with placebo dosages since you have no way of backtracking which patients had active administered and which had placebos. There's no way of deconvolving a placebo effect.

In the best case, positive responses are scaled down by the fraction of placebo used. It's not the way these challenge response tests are run and results can be particularly misleading since (in the AV world) a positive response for all placebos (i.e. genuine false positives) would be inappropriately interpreted as "better performance". You can apply statistics all you want, it won't fix starting with bad mathematics.

Blue

 

Anyway, I don't see why AV testers should absolutely adopt an exhaustive approach as described by IC. What is lacking, along with tester skills, is a sound statistical model. I do not see the reason why AV testing would not obbey to the same rules that govern other statistical testing/surveying practice.

- Define, a set of relevant criteria to classify malware. Such criteria could be for example: file type/subtype (binary executable [PE, ...], script [javascript, ...]), malware type (virus, backdoor, trojan horse), date (first seen/last seen ITW), mode of propagation, source (honeypot, ?).
- Quantify (roughly) the proportion active malware belonging to each of these category (prevalence). Optionnally: assign to each category a dangerosity factor (risk of outbreak ? Of information leak ? etc.).
- Weed out as much bad samples from your (full) collection as possible using automated tools.
- Classify the samples of your collection in these categories using all the information at your disposal.
- Pick "randomly" a representative subset of your collection (i.e. with the correct prevalence profile). The minimum size of the subset depends on the number of criteria and on the prevalence of each class.
- Analyze individually (disassemble, etc.) each and every malware of this subset. Count the number of "false positives", duplicates, and misclassified samples in each category. This gives an estimate of the error you can expect for the full set or any "representative subset" of it (function of its size).

Test either the full collection or a representative subset of it, and report:
- The risk associated with each antivirus (combination of prevalence and dangerosity)
- The expected uncertainty associated with the result (expected number of bad/badly classified samples), for each AV.

I'm convinced there is no need to disassemble/debug hundreds of thousands of samples to get a clear picture of the relative efficiency of the various antiviruses. Besides, at the cost of increasing the uncertainty, it could be possible to test those "HIPS" stuff proposed by most modern AVs.

 

I'd agree that there isn't an overwhelming need to follow IC's protocol - the payback for work expended is simply not there. However, the way I look at it, is that IC's approach is the only scheme that will provide reasonable assurance of a low noise result. In the absence of that, the results published by any tester will be subject to some level of intrinsic noise associated with the lack of fidelity in the testbed. "Flags everything reasonably suspicious" is a lot different than, for example, "flags 99% of malware".

Ultimately the problem devolves to that continually vexing question of whether a test result of 99+% is actually different from 96% or 90% or even 85%. This is independent of whether a given user is sensitive to 99% vs 85% detection based on their usage profile and the challenges that profile generate.

Blue

 

Just a note - finding and identifying viruses in a computer is not very important but stopping all viruses from executing on ones system is crucial. Some of the testing has been reported as "how many viruses did AV programme X find?" Not very important. How many known viruses did AV programme X stop from executing? Crucial.

Statistically speaking - one way some consider a programme's value is by how well has it done over a period of years. Has it stopped viruses from executing all of the time or part of the time? Are these statistics derived from lab tests or users experience?

In response to folks who have had no virus problems (even for years) who use either a) no AV or b) an inferior AV - perhaps you have not been challenged by any viruses so to say brand X worked fine for you is a serious mistake. It did not work at all. You simply did not have any viruses present themselves to your computer.

Just some food for thought.

 

Can you please explain to me whats the difference between Finding all viruses and identifying all viruses versus stop the executing of it? If you find all viruses and can identify the virus you are also able to stop them since you know it's a virus. That said: Detecting the virus in a on-demand scan will FOR SURE prevent the execution of malicious programs since the RTM (Real Time Monitor) will not even allow you to let them run and to infect your machine!

 

A computer may have a virus but it may not have been triggered to execute or is waiting for the sceduled time to execute. At this point it would not be a threat. If and when it executes it would spell very bad news for the compuiter. If it has not executed then there is no threat.

Yes we have heard it said that people do not want any malware of any type on their computer and it is a good thing to have a clean computer but cleaning is a much harder thing to accomplish. Operationally the main thing is not to have any malware execute on one's computer. There are Trojans which do not execute until a given time or after a given sequence so until it executes it is no threat. Viruses can be the same. If it has not executed it has caused no problem and will not until it executes. Simply having a virus on one's computer if it has not executed is no threat if one has the proper software to stop all viruses from executing.

Hopefully you have got it, right?

 

Nope. Because your writing makes no sense at all. If something activates later then it is still malware! You write that it doesn't represent a threat - of course it does. So finding this malware before you even install it is at least good as finding it when it tries to do its action.

 

If I read you right, you are refering to a situation where some malware has infected a system but has not delivered its harmful payload yet because it is waiting for an action/time/date.

If that is what you meant, I would say that the initial infection does represent a threat. Maybe behavioural detection could reduce the impact of the payload, but a threat still exists.

IMO, 'having a virus' means that the malware has been executed on the computer, which has had its configuration changed as a result. Maybe the malware will be able to deliver its primary, secondary or possibly even tertiary payloads. Maybe not and perhaps instead it will simply make the machine less stable. All of these situations are a threat to maintaining a system that keeps data accessible to authorised users and unaccessible to unauthorised users.

 

The software referred to of course has a library to compare for finding viruses and other malware before hand as well as superior heuristic capability. The thing is that if some virus has modified your pre-existing software then you have gotten infected. Using the software referred to has not allowed anyone using it (if not so then please post a link to documentation) to have a virus infect their computer. If the virus is or part of an executable and the .exe has not been executed then there has been no harm yet. Again if the virus has modified some existing software in your computer then it has made an infection. If the virus comes pre-packaged in some otherwise benign software then that software is already compromised and will either deliver a payload upon execution or be stopped by competent AV software.

If no virus attack then whatever else one wants goes beyond the realm of basic Anti-Virus software. We took a poll some time ago to learn if others here on this forum preferred a suite of anti-malware software or the layered approach having separate and different software handle each type of threat separately (i.e. Trojans, viruses, adware, spyware, worms, key loggers etc.). Most respondents wanted the layered approach but the majority of computer users worldwide would rather have a simple solution in a single package to manage all the malware.

So getting back to the veracity of Anti-Virus software testing in a lab - perhaps we can make an appeal to have people post their actual documented experiences to see if they match the lab results. What this suggests is that if someone has gotten an executed virus attack on their system while having AV brand X properly installed does their experience match the results of the lab testing and report? This could open up a whole new impetus for a new third-party non-profit website - to compare actual real-world experience to the lab results.

Meanwhile our computers have remained virus-free for many years. (The disclaimer here is that we cannot verify that any viruses had attempted to attack the computers as we do not do our own AV testing hence a clean slate). We made the decision to use this particular brand of AV software because of a serious lack of finding any verifiable documented evidence of any computer suffering a virus attack while using it.

It seems that anyone making a decision to use brand X AV faces one of these two scenarios:

Scenario One - The software in question is rather new and one must wait to learn of other user's experience or they can decide to trust some AV testing lab's findings for their decision or

Scenario two - The software in question has been around for awhile then an attempt to make a preliminary decision is made based on published reports and user's documented experience.

Either way trust is involved - either trusting one's own research to learn about the software hoping that enough pertinent information is found and/or trusting that an AV testing lab has done all they can to provide as solid of a report as can be done.

 

sure, my av finished bottom.

should i change my AV?... erm no is the simple answer

if the testing is accurate, av-test gave 92, av-comp gave 90 or there abouts, so an average of 91%, which isnt too bad anyway.

also, these percentages are over 800,000+ samples, nobody will ever get such things, i can count the number of viruses ive recieved on both my hands over the last 5 years.

so i could say, my av's .. whichever ive used, have had 100% detection

whenver i try online scanners or whatever, im clean... so it means nothing to me.

people sometimes forget this

 

C.S.J
Did you even get a virus attack or did your AV fend off all attacks?

 

yes my av has found and deleted threats.

 

how come others get all the fun ?

In 11 years of being on line I have yet to see a real live virus.
when I did run anti-virus programs I did get my share of false-positives which I religiously quarantined and then had checked so eventually I gave up and now only run on demand once a month or so.

So for me antivirus testing is corrupt because it supports the notion that without this or that program contamination is guaranteed within milliseconds of logging on.

Still it could be worse - I could always load up a HIPS program and wait for nothing to happen - only slightly less interesting than watching paint dry.

 

If you want to see viruses and malware, just start downloading .exe's in p2p apps, visit crack sites online and download keygens etc, and for good measure go to a lot of porn sites and wait for something nasty to appear in a java applet or something. It's out there, you just have to go find it..