How Security Products are Tested – Part 1

Discussion in 'other anti-malware software' started by Minimalist, Feb 27, 2017.

  1. Minimalist

    Minimalist Registered Member

    Joined:
    Jan 6, 2014
    Posts:
    14,447
    Location:
    Slovenia
    https://securelist.com/analysis/publications/77698/how-security-products-are-tested-part-1/
     
  2. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,400
    Location:
    U.S.A.
    Let's see if Kaspersky brings up the recent all "synthetic" malware tests used for testing "next gen" AI/machine learning solutions in the "Part 2" article.

    In regards to PC Mag malware samples, I believe they are now using samples provided by MRG which in turn gets those from AMTSO.
     
  3. Bill_Bright

    Bill_Bright Registered Member

    Joined:
    Jun 29, 2007
    Posts:
    3,844
    Location:
    Nebraska, USA
    The quality of the testing also depends on the source of the financial funding of that laboratory. This has always been a problem, regardless the products being tested. If funding comes from one of the vendors, there is at least the appearance of favoritism that put the results in question.

    This is why, in part, you can find a study for just about every browser that says this browser or that browser is best.

    Plus there is the objective of the program and the objective of the testing. Both can affect results. For example, many anti-malware programs are designed to score well on this test or that test so they can then claim to be #1 in that area. As itman correctly points out, those are "synthetic" scenarios that don't really represent the real world.

    Microsoft (who is not in the business of selling anti-malware solutions to regular consumers - you and me) announced they will not be developing MSE or Windows Defender to score well on laboratory synthetic tests, but instead to perform well to thwart today's threats. So MSE/WD don't do well in laboratory tests and that gives fuel to the makers and fans of the commercial products to boast about their products, and bash Microsoft's.

    The thing is, 100s of millions of users (including me) are using WD with no problems. So it must be doing something right yet you could get a different impression with the lab reports.

    My point is, I don't pay attention to laboratory anti-malware tests anymore because they are synthetic. They bombard synthetically setup computers with 100s of 1000s of so-called "threats", many of which are NOT circulating "in the wild" - therefore, are not threats after all.

    That article is a cleverly written promotion for Kaspersky - undoubtedly a good security program. And the fact of the matter is, most of the popular security programs out there are totally capable of protecting their users.

    But protection depends much more on just the security program of choice. Users should never, as in NEVER EVER rely on a single vendor for their security. Regardless your primary scanner of choice, users should have a secondary scanner for on-demand scanning to make sure the primary or the user (always the weakest link in security) didn't let something slip by. I use Malwarebytes for that.

    And regardless their scanner of choice, users should always keep their OS, installed programs and particularly the security programs updated and current.

    And lastly, users need to "practice safe computing". That is they need to avoid risky behavior like visiting illegal pornography and gambling sites, participate in illegal filesharing via torrents and P2P sites, and most of all, avoid being "click-happy" on unsolicited downloads, attachments, links and pop-ups. Again, these are all things we must do (or not do) regardless our anti-malware solution of choice.

    I like to say we don't need to drive around in an Abrams tank. We just need to keep our modest car fully updated and properly maintained, and we must drive defensively.
     
  4. cruelsister

    cruelsister Registered Member

    Joined:
    Nov 6, 2007
    Posts:
    1,519
    Location:
    Paris
    A few things:

    1). One topic that is never addressed is the age of the samples used in testing. Yes indeed one can find a plethora of script-kiddie knockoffs showing up daily, but the malware finding the widest distribution will be stuff that is quite fresh, often less than 12 hours old. Blackhats are not stupid and are well aware that malware older than this is prone to detection. So until the Testing organizations can promise that all samples tested are less than D+1 AND have wide distribution the results, although interesting as a comparison among products, may have little if any Real World applicability.

    2). Obfuscation methods- all too often this is done by some jive-time crypter. Yeah, you'll get a different SHA-256, but over the past few years this kind of crap does not fool any of the better AV's.

    3). Must be nice for some companies to claim to have some sort of simulator, then just post results without the possibility of independent verification but instead just say "Trust Me".

    4). If a product gives an alert for a brand new UNSIGNED application that happens to turn out to be legitimate, should this really be considered a false positive?

    5). As most reading this know already, some applications at default are sub-optimal, but by checking a box or two may be the Cat's Meow of Protection. One really can't fault a testing organization for not trying EVERY setting for EVERY product as the variables will multiply like insects, but the reader should keep this in mind.

    Conclusion- Testing reports may be fun to read if one is bored, but they never should be considered the Word of God.
     
  5. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,400
    Location:
    U.S.A.
    Most of the major AV Labs are AMTSO members. Here's a link to guidelines and best practices documents: http://www.amtso.org/documents/ . These documents are recommendations and nothing more.
     
  6. Bill_Bright

    Bill_Bright Registered Member

    Joined:
    Jun 29, 2007
    Posts:
    3,844
    Location:
    Nebraska, USA
    This is what I was getting at above. There is no need to test 100s or 1000s of malware samples from 10 or 15 years ago - especially if no longer in circulation. What purpose, for example, is served to see if an anti-malware program is able to block a threat designed to exploit a vulnerability in XP SP2? A vulnerability that was patched by SP3? Yet that one type of threat some labs still test for.

    Why should an antimalware solution that is integrated into Windows 10 by downgraded because it is not designed to block that harmless threat?

    I don't mean to sound like a WD fanboy. That is not my intent. No doubt WD is a "basic" anti-malware solution, just as many of the other integrated tools in Windows are basic tools. The integrated calculator is a basic calculator. The integrated WordPad is a basic word processor/text editor. The integrated Snipping Tool is a basic screenshot tool. In many cases a basic tool is all most users need. If you really need a more advanced tool, Windows lets you install it. But all I am saying is for most users the basic Windows Defender is all most users need, even though it does score highly on those synthetic laboratory tests.
    Yes. Because it is a false positive. But I don't feel in that situation, it should be cause to down rate the program. Who's responsibility is it to make sure a false positive does not happen? The security program developers cannot be expected to go out and buy or download EVERY new program and EVERY new version of those programs and test them against their anti-malware program. Or make sure every signed app is accounted for.

    So they must rely on the app makers to submit their apps for inclusion, and/or for their user base to submit those false positives for review.
     
  7. Ad1.
    It is funny how the in the wild test of different testing agencies find different numbers of NEW malware. I have an acquaintance in the security industry who says that even the best research labs only find 1 to 2 NEW samples a day (opposed to the claimed 30.000 new malwares found per day).

    Ad3
    To determine prevalence you should have overview of all malware, which is pretty much impossible, so we have to use statistics to define a representative sample set. Since we can't verify the statistical relevance of these sample sets and are we prone to the TRUST ME factor.

    Ad4.
    Agree I have set UAC to block elevation of unsigned since I started using Vista, without problems. Why allow unsigned software?
     
    Last edited by a moderator: Mar 1, 2017
  8. Bill_Bright

    Bill_Bright Registered Member

    Joined:
    Jun 29, 2007
    Posts:
    3,844
    Location:
    Nebraska, USA
    I have often wondered about that too. I used to work for a defense contractor who did software development for secure government networks. We had a team who did malware research. They were members of the various malware research organizations who shared such information. I too have heard of such reports of extreme numbers every day like 30,000. But much bigger numbers too, like 230,000! And 1 million new malware threats released every day!!!

    We kept asking who is reporting this? How are they coming to these numbers? And how can anyone defend against them? We decided it was mostly marketing weenie fluff trying to scare consumers into buying this anti-malware solution or that solution. Or to hire this security firm instead of that one. Or the marketing weenies for this research lab trying to outshine another.

    I am sure there are 100s, maybe several 1000, but nothing like suggested.

    It must also be noted that just because a piece of code is malicious, that IN NO WAY means it is even remotely capable of getting past the typical defenses already on networks and computers, like simple firewalls, routers, ISP scanners, spam checkers, simple default browser security settings, or operating system features or regular anti-malware scanners. It does not mean there is a vulnerability it can exploit. Nor does it mean it can deliver its payload if it some how manages to get past all defenses, or be able to "phone home" with any compromised data.
     
  9. roger_m

    roger_m Registered Member

    Joined:
    Jan 25, 2009
    Posts:
    8,072
    I also find it very hard to believe reported numbers on new malware.
    For example in Symantec's Internet Security Threat Report from April 2016, https://www.symantec.com/content/dam/symantec/docs/reports/istr-21-2016-en.pdf they claim that:
    On the other hand, Kaspersky claim that in 2016, 323,000 new malware files were detected daily.

    Personally I find both figures hard to believe. After doing some more searching on Google, I found this 2012 article from Ed Bott, The malware numbers game: how many viruses are out there?
    At the end of the article he says this:
     
  10. Thx @Bill_Bright and @roger_m

    Would be interesting when researches like AV-comparatives with a (I suppose) big enough network of honeypots and malware experts would comment on the claim of my acquaintance that despite the distrubution of maybe hundreds or thousends of slightly adapted malwares per day, they only find one to two really NEW samples per day.

    Regards Kees
     
  11. guest

    guest Guest

    There is a difference between brand new malware aka "0-minute" and its thousandth of variants...coming fews days after.

    Exact, most crafty blackhats create a new malware for a specific purpose (like targeting a specific network, individual, etc...) and use it only once or two times, until their goal is achieved. Then they resell it to kiddies.

    Yes.

    Security devs collect/receives checksums from the software devs.

    Anyway , to me, tests labs are just youtesters with a bigger wallet , their methodology are basically the same; very very few of them study each malware and its repercussions. They download batches of them and throw them against an AV... pointless...

    A discussion i created.
    https://malwaretips.com/threads/my-take-on-what-should-be-a-real-world-test.68352/[/QUOTE]
     
  12. cruelsister

    cruelsister Registered Member

    Joined:
    Nov 6, 2007
    Posts:
    1,519
    Location:
    Paris
    It really makes me want to (add curse word here) scream when I see the 30,000 or 30 million new samples a day drivel. As guest correctly states a single unique malware file may be morphed a thousand times and sold to wanna-be Blackhats. This doesn't make these variants either new or widely distributed. But the AV vendors have a vested interest in scaring the (add curse word here) out of you, making you paranoid enough to buy their products.

    Even worse are those (add curse word here) simulators which all too often are not seen as the mindless fun that they are but instead the results are viewed as the Word of God. For instance, consider the Keylogger tests. Do they ever state that blocking the transmission of stolen data is equally effective as detecting a keylogger hook? Do they say that although a product may stop clipboard theft it will be totally ineffective against a Pony or Banker trojan? Of course not- it's not in their interest to do so.

    Sorry for the rant- I'm getting myself pumped up for a video conference later today and some are in deep, deep trouble.

    And guest- SOME Youtube testers use really, really fresh stuff. Apparently method this is an acquired taste as passing a product using month old malware is much more popular,
     
  13. guest

    guest Guest

    i know who you meant by "some" , it is why i wrote "very very few of them" ;):D
     
  14. TonyW

    TonyW Registered Member

    Joined:
    Oct 12, 2005
    Posts:
    2,741
    Location:
    UK
    I do agree these figures are astonishing but I'm wondering if these numbers are based on the naming system they tag to every sample variant they detect. I note Kaspersky have point-letter appendages to each variant in their database and that would tally up a lot if you count each one rather than just individual malware family groups as one.
     
  15. illumination

    illumination Guest

    There are very few using fresher samples, and I agree, way too many using older samples. We know Cruel sister to not be one of the latter.

    guest is correct as well with variations of samples released vs new samples.

    Correct me if I'm wrong but I would assume that most vendors that store local databases will occasionally "clean up" the database of outdated no longer in the wild signatures to keep the database smaller/lighter for less system impact.
     
  16. guest

    guest Guest

    Correct, they will identify the common code of several strains of malware and create a unique signature (instead of a hundreds different one)
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.