NSS Labs files antitrust suit against multiple cybersecurity vendors

Come on! You know that is total nonsense! They clearly state on their webpage they test "Consumer Endpoint Protection" (Consumer EPP) products too! Why are you doing this? If your claim were true, why do they test common consumer anti-malware solutions? Why do they test common browser solutions? Why are you arguing against a company that only wants to provide you more information about security products so you can make informed decisions for you? Do you own stock or work for one of these companies?

And so what if they test enterprise and SMB security software? Enterprises are consumers too.

So NO! I am not kidding! Are you really suggesting Norton, McAfee and the others represent the interest of consumers? They NEVER have! Does AMTSO? Obviously not.

NSS Labs goal is to, as stated here,

They want to do that through testing and evaluations. Why are you opposed to that? I don't understand your motives to stifle information consumers (big and small - including you!) can use.

Huh? So now NSS Labs is a competitor of Norton, McAfee, and the others? No! Why are you now trying to obfuscate the issue with that innuendo? That is totally irrelevant to this discussion!

More obfuscation and also totally irrelevant to this discussion.

Well now that's the issue, isn't it? So how, itman, are consumers supposed to know which product is best able to "protect the OS and app software along with the resultant data" on their computers? Trust Norton? McAfee? ESET because they say so? Bullfeathers!

To protect and help consumers determine what is best for them! Maybe there should be a law (or court ruling! ) that allows for the exception of testing, evaluating, and publishing results of those tests. What a concept, huh?

Once again, the anti-malware industry has absolutely no financial incentive to defeat malware once and for all! They need malware to survive and thrive in order for their businesses to survive. Consumers have a right to know which product works best for them.

 

Securityweek.com just published an article that is pretty in depth on this issue:

Testing Firm NSS Labs Declares War on Antivirus Industry
https://www.securityweek.com/testing-firm-nss-labs-declares-war-antivirus-industry

 

I also can across another interesting posting in regards to fairly recent spat between NSS Labs and Cisco over testing restrictions:

NSS Labs-Cisco spat raises licensing restriction enforceability
https://searchnetworking.techtarget...t-raises-licensing-restriction-enforceability

And, this article does gets into the legal issues that certainly appear to me to be directly related to this NSS Labs anti-trust law suit; namely Consumer Review Fairness Act of 2016 .

This is also not the only recent history publicly commented upon criticism of NSS Labs and its testing methods by non-AV related product vendors.

 

I found that SecurityWeek article fascinating. Thanks. I found it clearly illustrates NSS Labs concerns and there does seem to be some sort of conspiracy going on.

I note where several of the EPP vendors prohibit comparative testing that may help consumers decide which is best for them.
VirusTotal agreed to restrict access to its malware database unless the vendors and testers joined AMTSO.
CrowdStrike sued NSS Labs claiming NSS Labs obtains their products for testing through "fraudulent means" but does not state those means. (CrowdStrike lost the suit).
Yes, NSS Labs wants us to pay for their reports -but note that is because AMTSO and the vendors prohibit public publication.

And now we see Cisco refused to activate software NSS Labs "purchased".

All this because NSS Labs feels they should not have to use AMTSO's testing methods.

I am still with NSS Labs on this. Perhaps not in the methods they are using to get their points across. But I believe they have valid points.

 

It's sort of in line with what I figured. A company can put whatever it wants in the EULA, but that does not mean that it is enforceable. I'm not sure where NSS labs is located, but the issue gets even more muddy if you're considering cross jurisdictional legal issues.

 

Appears a key point was missed in the posted NSS Labs vs. Cisco article:

As I stated previously, there is no law that can force a vendor to sell his product to a purchaser.

I really expect in the near future, Endpoint software licenses will not be auto issued once payment is rendered. Rather the license key/s will be issued via e-mail once the AV vendor verifies who the purchaser is. If the purchaser is an AV Lab it does not want to test its product, it will refuse to issue a license key and the purchase price will be refunded. As such, EULA restrictions on use will be a moot point although I still believe they are enforceable in regards to use of the software in public tests where the results are published w/o the AV vendor's consent. This also might have to be resolved in U.S. court in regards to if this activity is covered by the 2016 Consumer Protection Act.

Also and very important to note in regards to the 2016 Consumer Protection Act. You can still be sued for libel for anything that you publicly disseminate:

The above was also the issue in last year's Cylance vs. Sophos spat. As previously mention, the Sophos VAR acquired a copy of Cylance by misrepresenting that it was affiliated with Sophos.

 

That's so fishy I can smell the stink from here.

I agree that vendors, stores, and service providers have the Right to refuse to sell or provide a service to anyone. But in this case, Cisco already sold the product to NSS Labs. A purchase (which is a contract, BTW) was already conducted. What other industry can force a buyer to sell back something they purchase (other than for a government mandated safety recall)?

I am 200% behind a software developer's right to protect their IP (intellectual property). So I am a staunch defender of a software company's right to prohibit, for example, the transfer of OEM licenses to new computers, and in their efforts to stop piracy and theft. But this is no where near the same thing.

NO WAY! NO WAY should any consumer be forced give any commercial company their real name. Not even Microsoft goes that far! In fact, Microsoft and Windows goes to extremes to protect our real identities!

The software companies already have measures in place to ensure software is authentic. Consumers must enter a unique product/authentication "key" to unlock the software. If the software company has not taken the necessary steps to ensure only 1 key is being used, that is on the company.

What do you propose next? That we, as consumers, must show proof of identity every time we buy a Big Mac? That we sign a contract promising we will never publish a bad food review, even if the food was cold, moldy and gave us botulism?

Look at the second line in my signature! I fully accept we all must make sacrifices to protect our freedoms. But it's "We The People" for a reason!

 

I guess you missed my reference was directed to endpoint AV software that is used by commercial concerns. And, commercial software licenses are different than retail ones.

Also I am not personally endorsing anything. I am just stating the facts. If you don't like the current situation, complain to your elected representatives; they make the laws after all.

 

I run a small business. I know they are different. I can buy things for my business using my business account, or my personal account. Or I can even pay cash. But not really the point here.This is about consumer rights.

It seems you are throwing every thing into the mix in your attempt to justify AMTSO dictating how NSS Labs runs their business and how consumers get their information.

 

Oops. Sorry Peter. You jumped in there while I was typing.

I see no point in discussing this further so I will yield to your call.

 

Hmmm, I note the article points out Symantec is not denying the flaws NSS Labs uncovered. Symantec is just complaining how NSS Labs reports it. So in that respect, NSS Labs is right and those companies are actively (either conspiring together or independently) trying to stop consumers from learning about deficiencies in those products.

There's clearly fault to go around but as a consumer, I want to know which security product works, and which has flaws. So I vote on the side of the consumer and NSS Labs seems to be in that corner, even though they seem to be (if you believe Symantec and the others) a bit unethical how they profit from being in that corner.

This will be interesting to see how the court lines up on this one. I hope for the little guy.

 

As far as the recent article goes, Symantec is belaboring the issue of a product being tested without the vendor's permission. So if the court allows discussion on that, I can't see how it won't open up EULA agreements and specific prohibitions against this type of activity. Also what Symantec is implying is NSS Labs is engaged in extortion like activities by requiring tested vendors to pay for private testing which would allow them to correct any deficiencies found prior to NSS Labs public comparative testing. -EDIT- Also this deviates in practices of the other major AV labs. Here a vendors pays to be included in the comparative test. If they don't pay, they are not tested and not included in the comparative results. So I am sure this will be presented in the court proceedings.

Finally as noted in the article, none of the above addresses the antitrust collusion activities alleged by NSS Labs. As far as this goes, AMTSO's claim to immunity is:

If the court upholds this view, I can't see how it can hold the other defendants in violation since they were only assisting in the developmental efforts as participating AMTSO members.

 

Or as NSS Labs might claim, they were conspiring to rig the tests so their scores looked better than they really performed!

Again, I don't see why or how the anti-malware industry can dictate how their products are tested and compared, as long as the competing products are tested on a level playing field using the exact same testing parameters. It would be like Porsche demanding their cars only be tested on an oval track, and not allowing testing on a drag strip or mountain road. Why not?

Or it would be like a wireless keyboard maker only letting you test up to 20 feet away instead of testing for maximum range.

As far as NSS requiring vendors pay - is that really that different from Plug Load Solutions requiring PSU makers pay to have Plug Load Solutions tests their PSUs? If they don't pay, they cannot display the 80 PLUS logo which is pretty much mandatory of the makers want consumers to consider their product. And not paying does not prevent other review sites testing for efficiencies.

What if Intel and AMD got together, or AMD and NVIDIA got together and decided how outside reviewer would test and evaluate their CPUs and GPUs and then attempted to block any reviewer from using some other benchmark protocol? Should that be acceptable to consumers? Of if Dell, ASUS, ACER and HP dictated how review sites compare their notebooks?

 

Some products can excel on one area and be weaker in another. And testing company can perform a test in a way that shows one weaknesses or their strengths. For example: one product is really good at detecting PUP. So if testing company includes a lot PUPs in their testing sample, that vendor would get better score. Even if all products are tested with same testing procedure, results could still be biased.

 

No! "If all the products are tested with the same testing procedures", that's the very definition of unbiased testing procedures. Biased results means something was unfair. If all products (assuming they were designed to do the same thing) are tested using "the same testing procedures", that is a totally fair test!

Extending your example, what if there was some policy imposed on the testing facility by a "consortium" representing those products themselves that prevented the testing facility from reporting PUP detection results? Instead, all they could report were the results of specified tests designed by that consortium to detect specific malware from a list of malware provided by that consortium. Would you be okay with that?

What if Products A, B and C scored well and were given passing grades based on those restrictive testing parameters, so you decided to spend your hard-earned money on Product B. Then next week your computer starts slowing down so you run another program (Malwarebytes or Product A for examples) and it detects your computer is swamped with 100s of PUPs Product B let right on through.

Would you be happy you wasted your money on Product B? What if you found out that testing facility knew Product B was a poor performer at PUP detection, but because the consortium representing those companies imposed those reporting restrictions on the testing facility, they could not publish those poor results? Would you be happy with that?

If I, as a consumer, looked at a review site for advice, and Product B was recommended only to find out later it could not stop PUPs, and the review site knew that but couldn't tell me because the product makers wouldn't let them, I would be livid!

To me it would be the same thing as Car and Driver not being able to report how well the brakes worked in panic scenarios on the cars they were testing because the car makers only allowed testing during gentle stops.

What if you lived in sunny Florida and you wanted to paint your house. And you were looking for a good paint where the colors would not fade from the sun. But the paint makers got together and dictated that paint testing facilities could only test paint that was exposed to the sun 4 hours per day, 3 days per week. Would you be okay with that?

There are two issues here.

1. The security software makers are dictating how NSS Labs test their products and reports those findings. We know this is true because that is what AMTSO is for. From the consumer's standpoint, that is wrong!

2. NSS Labs is [allegedly] forcing product makers to pay an exorbitant amount if they want favorable "public" reviews. If true, that is wrong. ​

But these are two separate issues. Blocking NSS Labs in the courts from forcing the makers to pay should not stop NSS Labs from publishing their testing results, as long as, the tests are unbiased - that is, all products are tested using the exact same testing criteria.

 

Appears you missed @Minimalist point. It not a question of the methodology employed but of the samples used in testing.

If one reviews AV lab public reports, they all state or post references to the methodology used. None however have any details on the samples used. At most, some labs will show a graphic breaking down by malware category samples used. In other words, you are to assume that samples used are legit real supposedly in-the-wild malware/PUA's/PUP's, etc.. This NSS Labs "sink" traces back to its creation of, and being gracious, synthetic lab created malware to supposedly test Next Gen solutions detection capability.

 

I don't think I missed the point at all!

Yes it is! He specifically said, "the same testing procedures" - his words, not mine.

Are you suggesting testing "procedures" is not the same thing as testing "methodology"?

If the same samples are used to test every product, and if every product is tested using the same "methodology", how can that be biased?

 

That is the point.

Each lab collects its own samples for testing. AMTSO has a malware sample database which it appears no lab uses since the samples are not that great and "stale." AV-C commented on this in a past Wilders posting. Also what separates one lab from another in its testing efficiency is the samples it collects. If all AV labs used the same methodology and samples, there is no need for multiple AV labs …………. Remember that AV labs are not non-profit entities. They actually compete against each other for AV vendor test business.

 

Exactly! Now you are really confusing me. You just agreed to everything I've been saying all along!

Each lab should be able to set their own methodology and use whatever sample database(s) they choose. As long as each lab applies that methodology and uses the same database(s) on each product they test, that is fair and unbiased. And more importantly, it gives us consumers more than one review/test to look at.

Would you buy a graphics card or PSU based on just one review by one review site that was forced to use one set of test parameters that were established through a joint venture of NVIDIA and AMD?

 

It is also necessary to differentiate NSS Labs from the other AV labs most are familiar with. To begin with, it has an entirely business model than the other labs.

NSS Labs generates its primary revenue from selling very expensive security test reports to Enterprise customers. It's primary test source emphasis is on expensive network perimeter protection appliances; i.e. hardware based intrusion detection devices and the like. Its additional testing concentrates on Endpoint and server security software.

NSS Labs unlike the other AV labs performs dynamic malware testing. It does this through its large OPSWAT based server platform. View OPSWAT as one huge honeypot whose sole purpose is to entice malware to attack it. Devices being tested are attached to this platform and security software being tested are installed on the OPSWAT platform. OPSWAT runs on a 7 day, 24 hour continuous basis. At select time intervals, usually quarterly, protection results are compiled and then published as reports. These reports are available on the NSS Labs web site for purchase. The price range is from a few hundred dollars to thousand dollar plus depending on the retail purchase price of the security entity being tested. Bottom line - as far as malware capture capability goes, nothing can beat OPSWAT.

Obviously, this setup is an expensive operation to run. As such, AV vendors who wish to have their software tested are charged accordingly for this. This charge also entitles them the privilege of seeing the results and performing corrective actions prior to the results being complied and published. Those who do not pay are not afforded this privilege. However, NSS Labs makes it a point to test all the major AV vendor endpoint and server based software whether the vendor agrees to being charged for the testing. And, all vendor's results are included in the published test reports. Hence, the gist of Symantec's claim against NSS Labs.

 

No it isn't. PCMag reviews security software. Does that mean we have to differentiate them from other testing facilities? Note they even claim on their page they may get commissions through links on that page. What about ConsumerReports? PCWorld? Toms Hardware?

I disagree 100% with your premise every review site or testing lab should be lumped together with every other review site or testing lab - or else be differentiated (Alienated? Discriminated against?) from those in that "consortium".

Every single one should be totally independent and free to develop their own testing methodologies. And as long as they test each product with the same methodologies, it is a fair test.

No it doesn't. If the product is available to consumers today, it should be allowed to be tested and reported as is.

Now as a "common courtesy", giving the product maker a heads up may be the nice thing to do - especially if testing revealed a serious "zero-day threat" that may easily be exploited by bad guys. But an entitlement? For a product that has already been released to the public? No.

So what? As long as they do the same testing on each product, that is fair.

And FTR, dynamic malware testing is a good thing! It should be part of every labs testing methodologies. Static malware testing has limited effectiveness against sophisticated malware. (source)

It is clear, whether you see it, or accept it, that you are on the side of the anti-malware industry and not the consumer. I personally think that is too bad for two reasons. (1) Consumer rights and consumer protection keep getting trampled on right and left. And (2) The anti-malware industry has done a lousy job at protecting computers ever since they declared it was their job to do so way back when Microsoft wanted to put A/V code in XP, but Norton, McAfee and others cried to Congress and EU, "monopoly!", so Microsoft was forced to remove it.

It is the same as if you were on the side of insurance companies regulating the insurance industry. Banks regulating banks. Pharmaceutical companies regulating big pharma. Utility companies regulating utilities. Telecommunications regulating themselves. History has shown over and over again that never works and it is always the consumer who ends up with the shaft. Always.

 

If testing procedure is biased it doesn't matter that it's the same. For example - if testing company decides to test AVs while not being online or allowing it to communicate with their cloud, that will affect their results. AVs that rely more on protection from cloud would get lower scores. Testing procedure in this case is IMO biased as it was conducted under unlikely scenario for regular user, just to show one's weakness or strength. Since testing companies know where most AVs are good or bad they can easily modify their testing procedure to get desired results. So same testing procedure doesn't guarantee fair test.
That's why I think that it's good to have organization where both, AV vendors and testing companies, can discuss and decide about testing standards.