Dr Web and AV Comparatives

Innocent till proven guilty: hmmmmmmmmm

Let me state the facts of this entire thread then form your opinion.

2-AV vendors pull out of testing by AV-C, Reasons stated are inaccuracy of some samples and fees charged. As far as the first assumption no proof but only conjecture, that may have validity based on the expertise of the person posting.

Fees- AV-C totally within his rights to set forth any fee structure that he deems fair. Then it is up to the vendor to decide as to if the charge is equitable in their eyes and if they wish to proceed with being associated with AV-C.

This is really all this boils down to. You have 2 vendors who have chosen and have the right to not be associated anymore with IBK. You still have some major players who currently at this time are obviously satisfied with testing and pricing structure.


Did I leave something out. Oh yes, all the crap in between.

 

Dr. Web was one of the 8.
I say we start another thread.
"Dr Web and AV-test.org"!

 

Probably the most interesting thread i have ever read in this forum (AV section). Thank you all!

 

IMHO, WE, here, are in no position to judge the accuracy or inaccuracy of the samples. Believing either side, is only a matter of who you trust, IF you trust any of the two parts. WE, never seen the samples nor have we tested them. AV-C, says the samples are good, so uses them in tests and gives the results. That's what WE read. 2 AV-experts, say there are some garbage files, that WE can't see either.

Without being in position to verify either claim, you can only stay neutral or hope you place your trust to the right person.

 

I trust myself

 

Wise man.

 

you nailed it, I had vundo on my wife's pc, 3 times I thought it was gone 3 times it cam back, 5 programs later I'm still not sure...

 

Because Frisk stopt using the services of AV-C, where can I find a test that meets the "standard" of Frisk?

 

yes, im sure the top-scorers just recieve the dvds from IBK and just add them all to their database.

20 minutes work, simple and scores high in the test.

 

ive heard the same comments from drweb too.


i could quote the rest, but it seems almost offensive

 

I am not capable of testing a security application. Therefore, I rely upon tests conducted by the various "organizations." If a vendor's product is not tested, then I have no idea as to its capabilities.

I trust AVC more than any others, but would consider other test organizations, and would look at how the product performs among the various testers.

If I could not see fairly consistent test results by several testers I would not even consider purchasing the AV.

Although I do not trust the magazines to do objective tests, if results among them were generally consistent that would be enough. I bought my first AV, Bit Defender, as a result of PC Magazine's test results, and did not go wrong.
I generally ignore magazine rankings.

My conclusion is that nearly all of the AVs are plenty good enough for those of us who are safe surfers. I suppose it is possible that some attacker could get by some of the lower rated AVs, but I really doubt that it will happen. I guess ignorance is bliss. Although I use one of the best, I would not be worried if I used AVG, Avast, Dr Web, or F-Prot along with a software firewall, but would want to add something like SAS

The various arguments as to the validity of the tests, and the vendors objections are not helpful to folks like me. I want to see tests anyway.
I am glad that there are those here who have the skill to test, and assess the various tests, but I am not one of those and will continue to rely on tests in which I have some degree of confidence.

I think it is a mistake for the vendors to opt out of tests they do not like because they receive lower ratings. There are always excuses that can be used to justify opting out. But in the end it is "Out of sight, Out of mind."

Regards,
Jerry

 

Is there an "AV testing/ratings game" to be played, I wonder?

Game activities -

1-Tester accumulates large database of stuff, some of which is malware & some of which is (uhhh) "crap."

2- Tester tests AVs then sends their proponent companies "missed" samples (some part of which is "crap?").

3- AV company "A" plays the game & makes (computer-generated?) signatures for ALL the missed samples (including the "crap?"). AV company "B" doesn't follow suit.

4- On the next test, company "A" scores high; company "B" scores mediocre.

In a 2-player game (A-type companies versus B-type companies) I suppose "A" wins. However, there is a third player in the game -- namely, the people (like me) who buy A-type AVs based on tests. Do we win or lose by betting on the game's outcome, I wonder?

"Consumer Reports" does not charge fees to products in order to test them. Instead, "Consumer Reports" charges fees to the people who want to read their test reports. Is there such a thing for tests of security apps? Could there be (that is, would enough people pay for independent reviews of software"?)

 

Previously i had written, that the sample must be representative and random.

Now add, the QUALITY CONTROL, which is all what this dispute is about. Who controls if the malware used in tests is real malware (assuming that the rest of the testing procedure is impeccable)?

Ideally, there should be intra-lab and inter-lab control. Meaning, files being analyzed locally to verify their malicious behaviour AND exchange of data with other testing labs or the vendors too, that will compare their results on the same malware samples. The comparison, with documented explanation of why something is malware or garbage, will show who is at fault and he will need to rectify his methods. AV vendors could even put on a verification procedure the testing lab, each giving a number of samples with true or false malware and after watching if the testing lab will have false positives or negatives. Meaning, trialing the capability of the testing lab to identify real malware from garbage. Of course each lab should provide documentation as why the malware missed to be identified correctly is what the vendor claims to be and not what the tester isn't right.

This is what happens in other professional fields, where testing is very serious job and mistakes, can cost very much, even lives.

Regards.

 

I must have read this reply wrong, sorry about that.

 

It's perfectly clear to me now, thank you.

 

Correct. They must at least set a basic standard as to what they are supposed to detect. One thinks that one sample must be detectable, the other thinks not, this is the infamous "research bias".

One can google his heart out about research bias to see what it means and how it is dealt with in other professions and can try to find the correspondent part in AV technology.

They must agree on what they are supposed to detect. Once they do, they must make sure there is no sampling bias , but real world representation of the type threats (this would need a statistical measurement apart and of course the results wouldn't be absolute , but with a confidence interval. This would make the sample pool statistically significative. Usually a confidence interval of 95% is acceptable. ).At the end of this measurement, you would have a theoretically statistically correct mix of the type of malware that exists in real life (ex. 40% trojans, 10% rootkits etc or even also according to the way they get delivered to the target). On what is representative sample of the single malware, i don't know how to judge that. I mean, in Spain, spanish writers must have a higher prevalence. In Russia the russian malware writers.
Of course a lab would have to pay a statistician for this job...

Then you have to deal with pre-analytical (sample preparation, AV correct setup), analytical (correct scanning procedures) and post-analytical (correct presentation of the results) errors.

Then you have to do a quality control , to be sure, your lab is detecting what others are detecting too and what all should be detecting, and not just phantomatic malware, just because an incompetent employee is making errors of judgement.

 

And I think they will eventually come back with things from both sides tweaked and improved.

 

Actually, there's a host of bias issues involved in these challenge tests, none of which are readily addressed, although I'd personally put selection bias as a likely more significant issue.

One pragmatic point to keep in mind is that the controlled standardization protocols that you describe are not operationally workable in a situation in which the standards evolve much slower than the real challenge they attempt to depict - unless you wish to test on how well you've done in the past. By the time the standard testbed of samples is validated, the results of the test are likely to be irrelevant.

That's not to diminish some of the caveats noted above, for they are real issues. However, readers should not lose sight that approximate results (be it a back of the envelop estimation, or an AV challenge test) can be quite useful as long as it's understood as approximate from the start and the results are viewed with that in mind. These tests are approximate, despite displaying significant figures galore.

Pragmatically, I realize there is a disconnect here since most readers are either congenitally unable or unwilling to appreciate that 95% and 99% detected might actually be the same result. Perhaps that range is a lot wider, say 85 to 100% are objectively the same result. I also understand that, from a marketing perspective, being on the "wrong" side of that nuance is a commercially losing proposition.

The only way through this is to provide result metrics that objectively portray the degree to which performance differences are measured. In a testbed containing x% disputed material (= genuine junk + "misclassified" samples + other), very simple arguments would show that results within ~ 2x% of one another have to be viewed as the same unless further analysis is performed. Even for very modest levels of disputed material, differences between results are quickly blurred. For example, by this argument, 6% testbed contamination would render all products examined in the last av-comparatives on-demand test, objectively speaking, indistinguishable.

Blue

 

This post was one of the most interesting I have ever read on this forum. That is until Inspector Clouseau (an employee of FRISK) made the following statement about Vesselin aka bontchev (also an employee of FRISK): “Vesselin speaks *NOT* official on behalf of FRISK Software International.”

So we have Vesselin and Severyanin (an employee of Dr.Web) who are directly challenging the validity of AV-Comparatives (“AVC”) stating the test set includes a lot of “crap.” Those are pretty bold statements.

I do not understand why Inspector Clouseau would try and separate Vesselin from FRISK. Also, of those qualified as experts, why do you not provide us with your opinion Inspector Clouseau.

I would also challenge Stefan Kurtzhals (who I believe works for Avira) to give his expert opinion on the quality of the AVC test set.

Frankly, if those who work in the industry and know how to dissect a potential malware are not willing to state their opinion, then this thread to me is essentially dead and will end up being another boring argument back and forth between amateurs.

Right or wrong, I respect Vesselin and Severyanin for having the courage to state their opinions. However, their opinions are a little suspect considering both FRISK and Dr.Web have both withdrawn from AVC. On the other hand, Dr.Web has essentially been making these claims for a couple of years.

I challenge Inspector Clouseau and Stefan Kurtzhals, or for that matter anyone else who works in the industry dissecting malware, to step forward and provide us with your analysis of AVC and it’s testing quality. It would be informative to hear the opinion of an AV expert who scores high on AVC.

 

I think most of what was posted today was pure,"skata."