Cylance vs. Symantec, ESET by AV-Comparatives and MRG Effitas.

Discussion in 'other anti-malware software' started by malexous, Apr 1, 2016.

  1. malexous

    malexous Registered Member

    Joined:
    Jun 18, 2010
    Posts:
    828
    Location:
    Ireland
    Two reports based on Real-World Protection Tests by AV-Comparatives and Exploit Tests by MRG Effitas.

    CylancePROTECT vs. Symantec Endpoint Protection:
    http://www.av-comparatives.org/wp-content/uploads/2016/02/avc_mrg_prot_2016_02_24_cyl_sym_en.pdf

    CylancePROTECT vs. ESET Endpoint Security:
    http://cdn5.esetstatic.com/eset/US/resources/docs/reports/avc_mrg_prot_2016_02_24_cyl_eset_exploit_only_en.pdf
     
  2. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    8,055
    Location:
    The Netherlands
  3. ropchain

    ropchain Registered Member

    Joined:
    Mar 26, 2015
    Posts:
    333
    Last edited: Apr 2, 2016
  4. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    8,055
    Location:
    The Netherlands
  5. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    2,969
    Location:
    U.S.A.
    I posted in other Wilders threads exploit test results by NSS labs showing that Eset scored the highest of all retail products tested. Eset's retail versions are essential the same as its Endpoint products as far as core protection features are concerned.
     
  6. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    8,055
    Location:
    The Netherlands
    Do they also test specialized tools like MBAE and HMPA?
     
  7. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    2,969
    Location:
    U.S.A.
    No.
     
  8. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    8,055
    Location:
    The Netherlands
  9. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    8,055
    Location:
    The Netherlands
    I just read the anti-exploit report. HMPA, MBAE, Kaspersky and Avast (to name a few) outperformed ESET. So the question is, who is the better tester, NSS or MRG. It probably also depends on the testing criteria and what type of payloads were used.
     
  10. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    2,969
    Location:
    U.S.A.
    Hands down winner ........... NSS Labs. Best comparison is the bush(MRG) versus the major(NSS Labs) baseball leagues.
     
  11. ropchain

    ropchain Registered Member

    Joined:
    Mar 26, 2015
    Posts:
    333
    @Rasheed187

    Unfortunately no single test truly audited mitigation / AV software with a bunch of custom exploits. Only running exploits from Angler is not really a great proof of the effectivity. Things are starting to get interesting if mitigation (and AV) software gets to deal with exploit variations of for example CVE-2014-4114 or CVE-2015-2545 or any other type of exploit that employs new exploitation strategies.

    In the end no single tool will survive a targeted bypass attempt.
     
  12. VoodooShield

    VoodooShield Developer

    Joined:
    Dec 9, 2011
    Posts:
    4,876
    Location:
    United States
    Machine learning and Ai can be a valuable malware classification mechanism, but in my opinion, it should never be used as the sole method of determining the maliciousness of a file, or as a scanner to scan the entire hard drive on end points. I have spent the last 6 months developing VoodooAi, which is VoodooShield's new machine learning / Ai malware classifier component, so I have become quite familiar with this technology. Overall, it is remarkably accurate and precise, and it seems to do quite well with super clean and super malicious files, but it can have a difficult time with certain files.

    This is just the nature of machine learning and Ai. The example I tend to use is machine learning and Ai as it is used in facial recognition... sometimes a sample (picture) just happens to look like Jennifer Lawrence, when it really is not her. This is what I have experienced with VoodooAi. Usually it is correct, or at least pretty close. But with all machine learning and Ai, sometimes it returns weird or unexpected results.

    So nothing is perfect... for example, if you consider the overall accuracy and precision of other technologies such as heuristics and behavior blockers on their own, machine learning and Ai will most likely have a better detection rate, on its own.

    So yeah, machine learning and Ai is great at detecting super malicious files, and usually it is correct, and it will probably be used extensively by a lot of the security vendors in the future. But it should be combined with other technologies to help determine the maliciousness of a file, and it should not be used on its own.

    Then again, if machine learning and Ai were perfect, or 99.99% perfect, everything else would be obsolete.. but trust me, that will NEVER happen.
     
  13. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    8,055
    Location:
    The Netherlands
    Well, home users are mostly targeted by exploit kits, so if you can block most of them, it's already good enough.

    Yes but that's the problem, it's hard to fully protect against kernel exploits. So detection techniques will remain important.

    Actually, I must say that I like the MRG reports a lot more. The free NSS reports are very basic. Also, FireEye wasn't too happy with NSS:

    http://www.crn.com/news/security/30...cize-nss-labs-testing-firm-defends-itself.htm
     
  14. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    2,969
    Location:
    U.S.A.
    .QUOTE="Rasheed187, post: 2578076, member: 19444"]The free NSS reports are very basic[/QUOTE]
    You get what you pay for ............... NSS Labs emphasis is on testing corp. security solutions.

    Yes. They weren't too happy that NSS Labs "bombed them" on a test report.
     
  15. ropchain

    ropchain Registered Member

    Joined:
    Mar 26, 2015
    Posts:
    333
    I am not talking about kernel exploits. ;)
     
  16. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    8,055
    Location:
    The Netherlands
    Care to explain? You probably talking about exploits that are not based on memory corruption? I believe those types can be stopped with stuff like anti-exe, HIPS and sandboxing. If the exploited app is running restricted, the payload (in-memory or disk based) can't do too much.
     
  17. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    8,055
    Location:
    The Netherlands
    The question is, who is telling to the truth, FireEye made some heavy accusations. But you can say what you want, the MRG reports are quite comprehensive. The only thing that I don't like is that they don't explain how certain malware samples or simulators were blocked, was it by AV heuristics, URL filter, behavior blocker, know what I mean?

    https://www.mrg-effitas.com/recent-projects/comissioned-tests/
    https://www.mrg-effitas.com/recent-projects/our-projects/
     
  18. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    2,969
    Location:
    U.S.A.
    What FireEye didn't like about the NSS Labs review is summed up nicely in this excerpt:

    That bring us to the really contentious thing about this test – on numbers alone FireEye really didn’t do that badly, detecting 95 percent of web malware, 96 percent of email malware and 93 percent of exploits, giving an overall detection rate of 94.5 percent and a zero percent false positive rate. Although this is below the roughly 98-99 percent scores achieved by most of its rivals, the real problem NSS Labs found with the FireEye systems was their cost-performance.

    This plots the total cost per Mbps protected against security effectiveness, which in the case of FireEye left its product with a figure of $427.85 (£280) against the highest-rated Sourcefire Advanced Malware protection costed at $231.86. In NSS’s assessment at least, Sourcefire simply offers more protection for every dollar spent than does FireEye.


    Ref:. http://www.techworld.com/news/secur...security-test-controversy-burns-anew-3510729/
    Bottom line - the FireEye product tested costs twice as much for slightly below average protection capability.

    Relating to corporate philosophy, the main difference between MRG and NSS Labs is nicely summed with the following excerpt from the CRN article link you posted:

    NSS Labs does not conduct a "pay-to-play" model of testing in which vendors must pay to participate, Phatak said, adding that the firm also got out of the certification business in 2009 to bolster the legitimacy of its tests. NSS Lab engineers conduct testing based on customer requests. If a vendor declines to participate in testing, NSS Labs will buy the appliance or software to conduct tests, he said.

    "In any test and every test that is published and made public we do not take a single penny from the vendors that are examined," Phatak said, saying the firm attempts to mirror the Consumer Reports testing practices. "Most of the money we receive to pay our bills come from enterprise clients, many banks and oil companies, who require an evaluation based on testing data without the subjectivity."
    Related to test lab capability, methodogy, and procedures, the "hands down winner" is NSS Labs.


     
  19. VoodooShield

    VoodooShield Developer

    Joined:
    Dec 9, 2011
    Posts:
    4,876
    Location:
    United States
    "Making sure that a payload cannot be executed or can only run isolated is indeed a good solution."

    Hehehe, sorry, I will butt out of your adult conversation ;).
     
  20. cruelsister

    cruelsister Registered Member

    Joined:
    Nov 6, 2007
    Posts:
    977
    Location:
    Paris
    I find it highly curious that even though MRG states that vb scripts were used for the in the Wild testing SEP still scored highly (and apparently as an Unmanaged Client!). As a few major retailers found out in very highly publicized breaches (and some Banks found out in non-public breaches) SEP can be bypassed without much issue by vb based worms.
     
  21. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    2,969
    Location:
    U.S.A.
    You have to understand how MRG does its testing:rolleyes:. In this case, all they cared about was exploit prevention.

    We used this scoring for the following reasons:

    The scope of the test was exploit prevention and not the detection of malware running on the system.
    •It is not possible to determine what kind of commands have been executed or what information was exfiltrated by the malware. Data exfiltration cannot be undone or remediated.
    •It cannot be determined if the malware exited because the endpoint protection system blocked it, or if malware quit because it detected monitor processes, virtualization, or quit because it did not find its target environment.
    Checking for malware remediation can be too time-consuming and remediation scoring very difficult in an enterprise environment. For example, in recent years we experienced several alerts that the endpoint protection system blocked a URL/page/exploit/malware, but still the malware was able to execute and run on the system. On other occasions, the malware code was deleted from the disk by the endpoint protection system, but the malware process was still running, or some parts of the malware were detected and killed, while others were not.
    •In a complex enterprise environment multiple network and endpoint products protect the endpoints. If one network product alerts that malicious binary has been downloaded to the endpoint, administrators have to cross-check the alerts with the endpoint protection alerts, or do a full forensics investigation to be sure that no malware was running on the endpoint. This process can be time and resource consuming, which is why it is better to block the exploit before the shellcode starts.
    •Usually the exploit shellcode is only a simple stage to download and execute a new piece of malware, but in targeted attacks, the exploit shellcode can be more complex. For example it can be used to leak information about the victim.
     
  22. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    8,055
    Location:
    The Netherlands
    I read about the exploits that you mentioned. Like I said, if the exploited apps were running sandboxed and child execution was blocked with AE or HIPS, those exploits should have been mitigated. BTW, you probably talk about this stuff (see links), what do you think about Morphisec? They seem to be using a completely new approach to anti-exploit.

    http://casual-scrutiny.blogspot.nl/2016/02/cve-2015-2545-itw-emet-evasion.html
    http://blog.morphisec.com/exploit-bypass-emet-cve-2015-2545
    http://www.morphisec.com/how-it-works/

    Have you ever read a paid report, since you think so highly of them? All I'm saying, is that the MRG reports are quite good and most of them are free for end users.
     
  23. cruelsister

    cruelsister Registered Member

    Joined:
    Nov 6, 2007
    Posts:
    977
    Location:
    Paris
    ITMan- this test was actually two parts- the exploit prevention as well as the in-the wild test. And for the in the wild section:

    In this specific test, we used URLs that pointed directly to malware executables; this causes the malware
    file to be downloaded, thus replicating a scenario in which the user is tricked by social engineering into
    following links in spam mails or websites, or installing some Trojan or other malicious software.
     
  24. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    8,055
    Location:
    The Netherlands
  25. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    2,969
    Location:
    U.S.A.
    Correct. And, appears AV-C used their standard 0-day malware test bed for this specific analysis. From the quote you posted, it states that the URLs pointed to malware exe's. I assume these were executed via some type of script but in reality, it is unknown if this was indeed the case.

    It is interesting that in the exploit testing, it was stated that javascipts were one of the delivery means.

    The real point of interest in why Symantec allowed any AV-C testing of their product since they had a major falling out a few years back with AV-C testing of their Norton products?
     
Loading...