NSS Labs files antitrust suit against multiple cybersecurity vendors

Discussion in 'other anti-malware software' started by ronjor, Sep 19, 2018.

  1. ronjor

    ronjor Global Moderator

    Joined:
    Jul 21, 2003
    Posts:
    162,650
    Location:
    Texas
    Sean Lyngaas Sep 19, 2018
     
  2. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,591
    Location:
    U.S.A.
    Oh, my.

    NSS Labs in its recent Next Gen testing episodes of the last couple of years has shown that "it doesn't play by the rules" when it comes to lab testing. Now "it is crying Wolf!" It will be interesting to see how far this suit gets in court. All the AV vendors named appear to have plenty of "counter claim ammunition" based on publicly available data on these episodes to fully counter NSS Labs claims.
     
  3. Bill_Bright

    Bill_Bright Registered Member

    Joined:
    Jun 29, 2007
    Posts:
    4,040
    Location:
    Nebraska, USA
    Oh? How so? Please explain. Got links?

    And what rules? Who established those rules? The anti-malware program makers? If so, then good! I am glad NSS Labs don't play by those rules. If the anti-malware industry did not make the rules, then again, who did? And who says testing labs have to use them?

    To me, not playing by the rules would be bad IF NSS Labs did not use the same testing and scoring criteria on all of the test subjects. But if all the test subjects are on the same playing field and judged using the same criteria, then I see no problem.

    What I see as a problem is rules set by others that all testing labs must comply with. That would do two things. (1) It would tell the bad guys what to look for and (2) it would tell the anti-malware program makers what to code so they can score better on those tests. Neither would be good.

    What I find very interesting in these laboratory tests (which ALL are synthetic, BTW, regardless how strenuously the labs deny that) is the ones complaining the tests are faulty are those who didn't score well.
     
  4. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,591
    Location:
    U.S.A.
  5. guest

    guest Guest

    As is said, vendors see labs as marketing proxies, when those labs don't go their way, they bite back.
    So check which labs make vendors happy, then it is probably a useless one...
     
  6. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,591
    Location:
    U.S.A.
    I'll try to shed a "bit more light" on what is going on here.

    AMTSO has been named a co-defendant in this lawsuit. For those not familiar with AMTSO, it is best described as a trade organization tasked with creating best practice standards for the testing of AV software. Its members include AV labs such as AV-Comparatives, AV-Test, SE Labs, Malware Research Group, etc. along with most of the established AV software vendors. Here's a link to its membership list: https://www.amtso.org/members/ . Of note is NSS Labs is an AMTSO member.

    The main thing to note about AMTSO is that is has no disciplinary powers per se. In other words if a member choses not to adhere to AMTSO testing guidelines, it can't be sanctioned by AMTSO as far as I am aware of. By sanctioned I mean it is expelled from AMTSO membership or it's membership put in a suspended status. Per the prior posted link reference:
    This all relates to last year's AV lab test "debacles" to "accommodate" the upstart Next Gen software vendors though the creation of synthetic malware samples the Next Gen vendors demanded. Again, there are plenty of web references to this of which NSS Labs and AV-Test were the primary "perpetrators."
     
    Last edited by a moderator: Sep 21, 2018
  7. Bill_Bright

    Bill_Bright Registered Member

    Joined:
    Jun 29, 2007
    Posts:
    4,040
    Location:
    Nebraska, USA
    Right. As I and now guest have noted, those who score poorly are the ones doing the complaining. Your example just confirms that.

    It goes back further than that - probably back to the very first AV comparison tests. But it really blew up a few years ago when Microsoft Security Essentials and later Widows Defender started scoring well AND when users started realizing they were NOT getting infected with MSE/WD - especially WD in W10. That's when all the Microsoft bashers went into a frenzy over lab results, fueled by anti-malware vendors who depend on consumers' and advertisers money to survive. The very vendors who depend on malware thriving for their own existence need to score better on those tests so they could use those scores in their marketing campaigns.

    Think about that for a second. Only the commercial anti-malware vendors have the financial incentive for malware to succeed. If malware goes away, so do they. Microsoft is the only anti-malware solution provider that actually benefits if malware is defeated. For that reason, they don't really care about those "synthetic" scores! They care about "real world" results. So they code for "today's real-world threats". And despite what some may claim and want others to believe, WD works! If it didn't, 100s and 100s of millions of Windows 7, Windows 8, and Windows 10 users would be infected. But that ain't happening!

    I find it interesting you criticize NSS Labs for not playing by the rules when it is CrowdStrike, ESET and Symantec who are conspiring against the testing laboratories to force their rules on everyone else! In other words, it is CrowdStrike, ESET and Symantec telling the testing labs, "You will use our rules so we look good! And if you don't, we will have you blacklisted!" :(

    https://www.pcmag.com/news/363882/crowdstrike-symantec-eset-face-lawsuit-over-product-testin

    Note too the standard (from your link above) clearly states on page 4,
    So there is nothing in the rules that NSS Labs or AV-Labs must use Symantec's testing criteria. And that's a very good thing!!!

    "Perpetrators"? Wow! What a loaded, accusatory and, frankly very biased statement.

    Perpetrators? o_O NO! Victims? Yes!!! Along with us consumers.

    Calling them perpetrators is accusing NSSLabs and AV-Test committing illegal or evil acts!?!? And that act is not following the rules that allow Symantec, CrowdStrike and ESET to look good? o_O Wow!

    You need to go back and understand what that standard is for. It is NOT a law or regulation. It is not an industry standard like the ATX Form Factor standard that, for example requires ATX compliant power supplies output +12VDC, +5VDC, and +3.3VDC voltages within a tolerance of ±5%. Or requires ATX compliant cases to position PSU screw holes in specific places using specific thread counts.

    No anti-malware provider should be allowed to dictate how any lab tests their products. PERIOD. As long as the lab uses the same criteria for testing and scoring all products, the testing is fair.
     
  8. Minimalist

    Minimalist Registered Member

    Joined:
    Jan 6, 2014
    Posts:
    14,881
    Location:
    Slovenia, EU
    Even if testing organizations have right to perform their test as they wish, AV vendors can't be forced to allow their software to be tested. If they don't wish to participate they have right to decline it.
     
  9. Bill_Bright

    Bill_Bright Registered Member

    Joined:
    Jun 29, 2007
    Posts:
    4,040
    Location:
    Nebraska, USA
    Not sure that is true.

    They sure don't have to volunteer their software for testing. But what is to keep NSS Labs, you or me from obtaining the product and testing it without permission? Pretty sure, nothing - as long as we don't publish any falsehoods about the product.
     
  10. Minimalist

    Minimalist Registered Member

    Joined:
    Jan 6, 2014
    Posts:
    14,881
    Location:
    Slovenia, EU
    It seems that they can prevent unwanted testing through EULA. At least that's how I understand this (from link in original post):
     
  11. Bill_Bright

    Bill_Bright Registered Member

    Joined:
    Jun 29, 2007
    Posts:
    4,040
    Location:
    Nebraska, USA
    Hmmm, that's sad. To me, that just illustrates how shady CrowdStrike, ESET and Symantec are. :( They don't trust their own products to stand up to scrutiny and comparison to competing products.

    That said, the fact they all have those terms in the EULAs is not evidence of conspiracy. There would have to be some proof they talked about it and all agreed to put that term in.

    But, regardless what the EULA says, I don't see how they can stop anyone from testing. Publishing the test results may be another issue.
     
  12. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,591
    Location:
    U.S.A.
    Appears you haven't been following the Next Gen vs. established AV vendor battle that has been underway for sometime.

    First, the NSS Lab test results last year that started the "stink" I referenced were presented at a high exposure security trade show targeted at enterprise and SMB installations.

    Prior to this episode was the "infamous" Sophos vs. Cylance test stink where Cylance was initiating legal action against Sophos for testing the product without their permission i.e. EULA violation. That one blew over when it was discovered that a Sophos VAR was actually the perpetrator and Sophos corporate was not aware of this.
     
  13. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,591
    Location:
    U.S.A.
    That's exactly what the current lawsuit is about.

    NSS Labs main revenue stream is by selling detailed test analysis on security software. And the reports cost $$$$. It is an anomaly when they post public comparative software test results and when they do so, the results are stated in generalities. You only get the full details by forking over $$$$ for the detailed report.

    The vendors being sued are simply stating to NSS Labs that they can only publish test results on their products if it agrees to AMTSO established and commonly approved test guidelines.

    I also believe that if the issue was testing of AV vendor retail products, there wouldn't be an issue. But NSS Labs only tests those "once in multiple blue moons." The endpoint products are the AV vendors "bread and butter" when it comes to revenue.
     
    Last edited: Sep 21, 2018
  14. Bill_Bright

    Bill_Bright Registered Member

    Joined:
    Jun 29, 2007
    Posts:
    4,040
    Location:
    Nebraska, USA
    Right! NSS Labs feels they should be able to test and publish the results. And I totally agree with that. Consumers have a right to know how those products perform - especially when they have to pay through the nose for them, and then pay over and over again for renewals.

    FTR, I sure am not suggesting users depend only on NSS Labs. They should look at all the review/testing sites.

    And it is not just about the full retail versions. Most of those free versions hound users to upgrade and/or hit them with ads, either for 3rd party products, or to upgrade, or if they are not careful, their home pages, and search engines are hijacked, or their browsers are loaded up with unwanted toolbars. :(
    But that's a bunch of BS! Why should the anti-malware makers dictate the testing parameters? Are the bad guys going to observe those rules? NO!

    Hardware review sites push hardware beyond published specs all the time. How is that different?
    Car review sites push cars beyond normal limits all the time to see how they react. How is that different?

    So? Do you think Car and Track or Consumer Reports publishes every detail? Of course they don't. They use little charts and a small generalized narrative.

    I fail to see why you don't want to see a wide variety of tests instead of the defined, narrow focus of just one standard that all testers must use. That makes no sense. Do you pay attention to just one review when you are considering a purchase?

    What is better for us consumers?

    It just makes no sense that products under evaluation only be evaluated by what ends up being puppets of the product maker. It not only makes no sense, it is just wrong. That's why we don't have the police policing the police. Its why civilians are in charge of the military.
     
  15. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,591
    Location:
    U.S.A.
    I was reading a few of the AV vendor's EULA.

    Avast definitely has a clause prohibiting publishing of test result's w/o their explicit permission. Ad hoc testers, you have been warned:
    https://www.avast.com/en-us/eula

    As best as I can determine, Eset's EULAs have no such restriction.
     
  16. Bill_Bright

    Bill_Bright Registered Member

    Joined:
    Jun 29, 2007
    Posts:
    4,040
    Location:
    Nebraska, USA
    I think it will be interesting to see how this lawsuit ends up. I don't think those prohibitions will stand up in court.

    At any rate, I think I will make it a point to NOT recommend any product with such a clause. Just checked the Windows 10 EULA. There is no such clause for benchmarking or testing. So good to go there.

    If ESET doesn't, then what is their complaint?
     
  17. Triple Helix

    Triple Helix Specialist

    Joined:
    Nov 20, 2004
    Posts:
    13,262
    Location:
    Ontario, Canada
    NSS Labs will not survive this even if they win IMO as they will hurt there reputation beyond repair. I take all testing firms with a grain of salt anyways so I hope they get hammered and lose badly!
     
  18. Bill_Bright

    Bill_Bright Registered Member

    Joined:
    Jun 29, 2007
    Posts:
    4,040
    Location:
    Nebraska, USA
    Wow! That's sad too.

    Who are you suggesting will look out for consumers and make sure these anti-malware providers, who have failed to protect us over the years, will not run roughshod over consumers and continue to milk many of us with exorbitant fees? Norton? McAfee? ESET?

    Yeah, right!

    FTR, I don't put total faith in ANY of those labs either. As I said, they can only "simulate" and "synthesize" the real world. But they can easily expose frauds and sadly, there many fraudulent security programs out there too. The are a necessary evil, IMO.
     
  19. Triple Helix

    Triple Helix Specialist

    Joined:
    Nov 20, 2004
    Posts:
    13,262
    Location:
    Ontario, Canada
    Well if you have been looking at tests from many sources over the years there are to many ups and downs for the well known anti-malware providers, but is that going to stop someone from using there favourite product? I don't think so...... :blink:
     
  20. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,591
    Location:
    U.S.A.
    AMTSO response to the lawsuit. The gist of the issue quoted below:
    https://www.amtso.org/amtso-rejects-baseless-allegations/
     
  21. Triple Helix

    Triple Helix Specialist

    Joined:
    Nov 20, 2004
    Posts:
    13,262
    Location:
    Ontario, Canada
    @itman no surprise here! Thanks.
     
  22. Cutting_Edgetech

    Cutting_Edgetech Registered Member

    Joined:
    Mar 30, 2006
    Posts:
    5,694
    Location:
    USA
    I will have to read into this more when I have time. I'm presently an InfoSec student, and AMTSO is one of only a handful of organizations we use for standards, and guidelines when testing.
     
  23. Minimalist

    Minimalist Registered Member

    Joined:
    Jan 6, 2014
    Posts:
    14,881
    Location:
    Slovenia, EU
    Thnx @itman for follow-up information :thumb:
    That doesn't show NSS Labs in good light. I wonder how court will rule about it.
     
  24. Gein

    Gein Registered Member

    Joined:
    Dec 8, 2013
    Posts:
    219
    I would think it would depend on the wording of the EULA, the jurisdiction, or even the judge ruling on the case to determine whether that particular clause or even the EULA itself is binding.
     
  25. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    17,546
    Location:
    The Netherlands
    I fully support NSS Labs, I'm sick and tired of security tool vendors that are too scared to participates in security tests. Of course products should also be tested in a fair and correct way.

    This is ridiculous, so you can get sued even if you're a non professional tester who puts videos on YouTube?
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.