Investigation in Progress by AV Comparatives

So, these are two products that participate in all three tests.
We can already eliminate some vendors from offenders' list.

 

This all makes me question the methodology of testing; using “submitted” AVs vs. download / purchase of commercially available products.
i.e: Consumer Reports when testing cars (or anything else) does not accept vehicles from manufacturers, rather, they go out to a dealer as a consumer and purchase them (to eliminate the likelihood that they may end up testing a “doctored” version).

 

"submitted" does not mean "send"; it only mean they tell e.g. which version to use (e.g. free, AV, IS, TS, etc.) the products are downloaded from official websites like you guys do.

 

Contrast that with a scenario where the vendor/developer is paying the tester to test their product(s).

 

http://weblog.av-comparatives.org/?p=501

 

I posted this in 2013:

And now my favourite reply from one of the many right-click-scan-tests lovers here at Wilders back then:

 

If you guys downloaded from an official site, how was it possible you got the rigged version? If it was on their website and available for public download - where's the problem? Was this a beta version? Not following how YOU got this version at all.

 

That's very interesting. Never thought of that.

 

AV-C do say in their blog that

so is it not possible that for a short time the 'rigged' version was publicly available to everyone, including the testers?

 

If I remember correctly it was only limited time download, removed after testing was done. But I might be wrong...

EDIT: TonyW beat me to it...

 

Then AV-C's testing protocol is faulty. What if all AV vendors started doing that? If they did AV-C wouldn't know whether to ~ Snipped as per TOS ~ or go blind. How many vendors actually have done this in the past?

 

That's the million dollar question, FF.

 

More or less everyone. If not this, something similar.

 

I am lost so you are downloading programs like we do? "Oh yeah av-comparatives downloaded our program. We can take the rigged program down." That seems a little hard to swallow.

 

Let's say I am an AV vendor.

<x> month: FDT testing starts. Time to reduce the sensitivity of my heuristics and bump up the signatures, add more samples quickly.
<y> month: Real-world testing starts. Time to "reconfigure" my cloud for a few days....

<rest of the time>: holidays for us folks, back to usual

Does this happen? Maybe yes, maybe no.... but this way, I do not need to release or submit and "special" versions of the software

 

If I am the CEO of one of the vendors and I know my company did not cheat, I want AV Comparatives to identify as soon as possible the guilty parties- if any.
I do not want my company name associated with flawed testing methodology or with cheating.
Certainly, I want the identities of those companies who cheated revealed so to exonerate my brand.
As of now, all participants are under scrutiny- deserved or not.
This indecision on the part of AV Comparatives is probably costing the honest vendors considerable revenue, and they have done nothing wrong.

 

At this stage of the game, unless AV-C is unsure of it's initial analysis, I can think of no justification whatsoever for AV-C not making public the name of the offending party. It does involve a breach of contract, but it is unimaginable that AV-C would agree to a non-disclosure agreement as part of any settlement with the offending company. If AV-C did that it might as well turn off the lights.

Unless there is a lot more to the story, AV-C appears to be handling this situation irresponsibly and unprofessionally. While it has no legal duty to disclose the cheater, it has some moral obligation to it's followers to do so as soon as possible.

 

How so? It's an investigation. You don't start dropping names right off the bat. You wait until you have the information and release the results at that time. Just because we don't know who is being investigated doesn't mean it's being handled poorly.

 

"AV-C has uncovered an infringement of the testing agreement by one of the vendors participating in its tests. It has been found that a product submitted for testing by the vendor had been specifically engineered for the major testing labs, including AV-C; public availability of this version was limited...."

"http://weblog.av-comparatives.org/?p=501

Fiction:

Statement by Cambtell Soup. "We are recalling all of our latest shipments of tomato soup. It has been contaminated. We are continuing to investigate this occurrence."

You don't sit on information and let peeps unknowingly buy from or use a product from a dishonest/unethical vendor if releasing the information you are sitting on can prevent it.

Frankly, I am stunned that any vendor would be dumb enough to do this.

 

In hindsight it might've have been better for AV-C to wait until their investigation was completed before making any announcement. Once the analysis and communications were finalised, a press release could be published along the lines of:

Following recent tests conducted in [specify when], we discovered vendor XYZ did...
We investigated and collaborated with [name of other testing groups] to verify facts...
The results of this investigation show...
We, the collective testing groups, have agreed to take the following action(s)...

No doubt they will do something like this, but by publishing the initial blog post it has produced speculation upon speculation here and elsewhere. I'm sure it's something they could've done without, but the fire has already been stoked.

 

Maybe somethins like this:

I know XXX test starts this month so I put a version with special optimizations on my website and update the common download link and wait for the lab guys to download.

 

That's what I would of thought was a better way to go.

 

That would be wrong. The fact is that they have clearly stated they have a identified a cheater. In the meantime, peeps are left to guess who it is. Peeps will be unknowingly buying or using a product made available by an dishonest/unethical/cheating company. (I am not assuming that the product available to the public is bad, though it is likely to be not as good as its test results indicate.) You can't let the cat half out of the bag. Meanwhile, honest vendors suffer as peeps put off buying decisions knowing that there is a rat in the market. Further collaborative investigation can continue after the cheater who has already been identified is made known.

The simple fact is that AV-C is being irresponsible by not disclosing the name of the cheating company. There is no way around that fact. There is only a partial argument for not releasing the name of the cheating vendor. To determine how AV-C 's procedures allowed this to occur and whether it was negligent and bares some responsibility for the failure, what procedures it can put into place to assure it does not happen again, and to re-examine all it's prior tests to see if this cheating also occurred in those tests.

Naming the vendor will not preclude a deeper investigation. What? It's unfair to the already known cheater to release it's name before AV-C completes an investigation to see if the second suspected cheating vendor actually cheated?? It's wrong to make known the name of the cheater until all testing labs discover how and every way and in which tests the cheating was done?

AV-C is unwittingly collaborating with the cheater by possibly harming current buyers in the marketplace. It is an accessory after the fact. If the police have proof that someone committed rape but also suspect another of doing the same, should they should wait until a full cooperative investigation with the FBI is completed in determining the full details of what the known rapist did to his victim and how many other rapes he has committed previously and completing their investigation of the second possible rapist before letting known to the public the name of the person they know has committed rape?? This cheater is currently raping the AV marketplace and harming other vendors and possibly harming those who buy or use the product.

AV-C has made a huge error in judgement . It has however possibly accomplished three things by its half-baked announcement. It has told it's followers to ignore the results of all of its completed tests (it does not say which test(s) the cheating occurred in) and that there is at least one and possibly two dishonest/unethical/cheating vendors loose in the wild, and that AV-C needs to review its procedures to assure that this does not occur again. AV-C is to be given credit for making itself totally irrelevant until the full investigation is completed and all the final discoveries are made public.

 

at least this has got folk talking about the testing used and whether these tests in the end mean anything,if tests are done to a known timetable it is relatively easy for the vendors to manipulate them,just look historically at the VB tests,eset didn't miss any examples thrown at it for years but in the real world it always offered less protection than kaspersky which at that time missed examples fairly regularly in the tests