In-the-wild Ransomware Protection Comparative Analysis 2016 Q3

Discussion in 'other anti-malware software' started by itman, Jul 22, 2016.

Thread Status:
Not open for further replies.
  1. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,591
    Location:
    U.S.A.
    https://www.mrg-effitas.com/wp-content/uploads/2016/07/Zemana_ransomware_detection.pdf

    Interesting test paid for by Zemana between ZAM, MBAR, HPMA, and CryptoPrevent. The winner ...... ZAM. Ten non-0-day ransomware samples plus a one synthetic python MRG created sample that no one detected.

    MRG contention that pre-execution reputation and sandbox execution scanning more effective at ransomware detection than post-execution behavioral analysis. Bodes well for most of the major AV scanners that employ heuristic analysis and reputation scanning coupled with internal sandboxing.

    Test details are that MRG changed the ransomware samples code so ZAM would not detect the non-0-day samples by hash signature forcing use of reputation analysis and cloud sandbox execution.

    -EDIT- Noteworthy is no script based ransomware tested which I believe ZAM would have failed on.
     
    Last edited: Jul 22, 2016
  2. guest

    guest Guest

    thanks a lot, very interesting the results of Zemana AM, is a must have right now
     
  3. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,591
    Location:
    U.S.A.
    Forgot Bitdefender AR that was also tested
     
  4. Aura

    Aura Registered Member

    Joined:
    Mar 19, 2015
    Posts:
    107
    Location:
    -
    I'll post the same thing here as I posted in the Zemana thread on BC: why include Malwarebytes Anti-Ransomware which is still in beta and expect it to compete on the same level as other products?
     
  5. FleischmannTV

    FleischmannTV Registered Member

    Joined:
    Apr 7, 2013
    Posts:
    1,094
    Location:
    Germany
    The exclusion of complete security products like KIS, EAM, Norton and the like, as well as the inclusion of beta software, is very suspect to me.
     
  6. guest

    guest Guest

    I guess there aren't many standalone anti-ransomware products to chose from.
     
  7. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,591
    Location:
    U.S.A.
    In the beginning of the test result report, Zemana set the test criteria; namely ZAM versus generic ransomware protection products.

    I agree it would be nice if one of the AV labs ran a ransomware comparative using major AV products. -EDIT- MRG did do a comparative here: https://www.mrg-effitas.com/wp-content/uploads/2016/05/MRG-Effitas-360-Assessment-Q1-2016.pdf that included ransomware samples(70). No mention on ratio of 0-day versus non-0-day. 14 out of 16 vendors scored 90% or better.
     
    Last edited: Jul 22, 2016
  8. haakon

    haakon Guest

    Wow... HMP.Alert. :thumbd:

    BDAR is limited to "CTB-Locker, Locky and TeslaCrypt crypto ransomware families."
    Source: https://labs.bitdefender.com/2016/03/combination-crypto-ransomware-vaccine-released/
    I'm not up on all the ransomware nuances, but there should be those resulting in failures to detect??
    Looks like the product worked as advertised, snagging the ones it was designed to. Its low score is disingenuous.

    Too bad WinAntiRansom wasn't represented.
     
    Last edited by a moderator: Jul 22, 2016
  9. Hiltihome

    Hiltihome Registered Member

    Joined:
    Jul 5, 2013
    Posts:
    1,131
    Location:
    Baden Germany
  10. daman1

    daman1 Registered Member

    Joined:
    Mar 27, 2009
    Posts:
    1,292
    Location:
    USA, MICHIGAN
    And also MBAR is still in BETA not equal playing grounds there.
     
  11. Techwiz

    Techwiz Registered Member

    Joined:
    Jan 5, 2012
    Posts:
    541
    Location:
    United States
    What a surprising coincidence :D
     
  12. Cutting_Edgetech

    Cutting_Edgetech Registered Member

    Joined:
    Mar 30, 2006
    Posts:
    5,694
    Location:
    USA
    I'm still reading the full report, but it's a shame WAR (WinAntiRansom) was not part of the test. WAR has outperformed other products specialized in detecting, and defending against Crypto-Malware in Cruel Sister's test. I'm willing to bet that it would have scored really well.

    Congrats to Zemana AM for doing well. Too bad they did not respond to my bug reports. I had to uninstall ZAM. It caused explorer to crash on my machine.
     
  13. FleischmannTV

    FleischmannTV Registered Member

    Joined:
    Apr 7, 2013
    Posts:
    1,094
    Location:
    Germany
    Excellent timing on their part :argh:
     
  14. RJK3

    RJK3 Registered Member

    Joined:
    Apr 4, 2011
    Posts:
    862
    Would have been interested in basic policy based protections like Cryptoprevent.
     
  15. markloman

    markloman Developer

    Joined:
    Jan 25, 2005
    Posts:
    581
    Location:
    Hengelo
    Yes, we released HitmanPro.Alert 3.5 yesterday and indeed, it makes the report outdated. But to inform the public, let me redo the Zemana sponsored report from MRG Effitas, but now with HitmanPro.Alert version 3.5.

    But before scrolling to the results, it’s important to know the scope of Zemana's test by MRG. Also, it’s important to know how most crypto-ransomware is delivered:
    • web pages (exploit kit)
    • weaponized documents (macro) attached to phishing e-mails (as illustrated below)
    • emails attachments (.js, .wsf, .pdf.exe, .scr, .exe in zip-archive)
    Example crypto-ransomware attack-flow:

    Ransomware-attack.png
    Note: Real-world delivery was not part of the test.

    HitmanPro.Alert (HMPA) offers e.g. Application Lockdown against weaponized documents, and Exploit Mitigations against exploit kits. These technologies block the malware before it runs or is even delivered on the machine. In addition, technologies like Process Protection, CryptoGuard and the new WipeGuard (part of CryptoGuard v4 in HMPA 3.5) offer protection for when the crypto-ransomware has become active. HitmanPro.Alert does not rely on signatures or the cloud.

    Zemana AntiMalware (ZAM) relies heavily on signatures and the cloud, like most AV/AM software. For the test, their software was extra configured to allow even more cloud communication. Also, over half of the samples in the test set were already known to AV labs and their engines for days, months, even years. These same engines are in use by ZAM in its cloud. You can therefor assume that the threat was blocked on signature from a third-party AV lab. This approach works in ZAM's favor as the test results show.

    The problem
    But the core problem with ransomware is this: you need to detect ransomware at the moment the threat starts to encrypt your files. Not a minute later! The encryption is often not reversible, unless you have backups.

    It is hard to get zero-day samples, as indicated by MRG Effitas. Even though the sponsor decides, I think MRG did a good attempt with what they had to work with. The test included different ransomware samples (thus techniques), but since most samples were already known to AV, the report does not reveal ZAM's ability to protect against new or unknown crypto-ransomware. The simulator should have (and does in a way) but seems to be completely different compared to any crypto-ransomware out there (the description in the report sounds valid but HMPA would've definitely caught it then and there because of the way it works). And it doesn’t matter that the hash signature was changed of the samples. It is only ZAM that is vulnerable to this as it needs to upload the file to its cloud again, where it is analyzed by third-party AV engines, who are not easily fooled by the changed hash signature.

    1 AV vs 4 non-AV
    It is also interesting that the sponsor told MRG Effitas who to include, that no other AV/AM was part of the test. So basically it was 1 AV vs 4 non-AV. I personally don’t think you can conclude that ZAM is specifically a good anti-ransomware, knowing that only 10, mostly old, threats were tested. The report actually says more about the non-AV in the test.
    HitmanPro.Alert 3.1 build 373, with CryptoGuard v3 (2015), seemed to score low but performed good and expected against the more prevalent threats. Build 374 was also publicly available 1 day after the test began, but I think it would not have affected the report.

    HitmanPro.Alert 3.5
    Most of you know that we’ve been working on HitmanPro.Alert 3.5 since December 2015. In all honesty, we should’ve upgraded our 3.1 users to 3.5 a lot sooner because, as the test reveals, even though most ransomware was blocked by CryptoGuard v3, you could still lose a file (depending on the disk access technique or the test documents/lures). Because of the scope and applied scoring method in the report, it draws a skewed picture about HMPA as we score mostly just 3 points (perhaps because of a TXT file that is attacked first). We apologize for not releasing our latest technology sooner. The new CryptoGuard handles ‘imperfect’ crypto-ransomware (like Mircop) and also has an improved rollback mechanism that restores encrypted files to their original state, incl. TXT.

    Results

    So without further ado, the HitmanPro.Alert 3.5 results:

    #1 CTB-Locker 2014-12-30 https://virustotal.com/en/file/8567...e462df681f4d4ea5bb7875148cb4ab25be4/analysis/
    Doesn't run, intercepted by Process Protection (Caller Check) of HitmanPro.Alert 3.5. Even with Process Protection disabled, blocked by CryptoGuard. Encrypted files are fully restored. Not a single file lost.

    #2 Petya 2016-03-23 https://virustotal.com/en/file/26b4...b05e94d43f3201436927e13b3ebafa90739/analysis/
    Intercepted by WipeGuard (part of CryptoGuard v4) which protects the Master Boot Record (MBR). Not a single file lost.

    #3 TeslaCrypt 2016-04-04 https://virustotal.com/en/file/d8ee...d18ee52569c046df74fa0dfe7e33d9ec422/analysis/
    Intercepted by CryptoGuard v4. Encrypted files are fully restored. Not a single file lost.

    #4 Cerber 2016-07-22 https://virustotal.com/en/file/85ca...2ed2a9370cc90a82b8586c88a259bb4a238/analysis/
    The sample in the report is different, 7b6c225989d2a1f1bd845fa620c1fc2e5196ab2673cca16a05b9929c152e7d65, from 2016-06-16, but its C&C appears to be offline now and no longer functions.
    This newer sample works and is intercepted by CryptoGuard v4. Encrypted files are fully restored. Not a single file lost.

    #5 Mircop (Autoit) 2016-06-22 https://virustotal.com/en/file/af84...b2f7655d52b6d4e66675efd7e3a1101d9b0/analysis/
    Intercepted by CryptoGuard v4. Encrypted files are fully restored. Not a single file lost.

    #6 Crypt0L0cker 2016-06-23 https://virustotal.com/en/file/67fd...ea41036008883ee57e52092257e6ced71c7/analysis/
    Intercepted by CryptoGuard v4. Encrypted files are fully restored. Not a single file lost.

    #7 Alphacrypt 2015-05-07 https://virustotal.com/en/file/99fc...985ab773b19c8cef8786ffc1fa50e35af29/analysis/
    Doesn't run, intercepted by Process Protection (Hollow Process) of HitmanPro.Alert 3.5. Even with Process Protection disabled, blocked by CryptoGuard. Encrypted files are fully restored. Not a single file lost.

    #8 ACCDFISA (Winrar based) 2012-02-21 https://virustotal.com/en/file/59ed...5be8a6e6b79dcf4f90a5c51f2bb12190bf9/analysis/
    HitmanPro.Alert 3.5 build 545 (or lower) does not offer protection against this threat. ACCDFISA stores your files in password protected RAR files and deletes the original files. In effect, it's only deleting your files on the disk and is therefore not picked up by CryptoGuard, which is designed to stop crypto-ransomware that encrypts your data on the disk. The data can be restored using one of the three static passwords (available on the web) or a free disk recovery tool like Recuva.
    Note: This old threat was in use in 2012. It was used by an attacker that targeted Windows servers through RDP (Remote Desktop Protocol). To be able to get onto the machine from remote, RDP needs to be exposed to the internet and the administrator account must be secured with a common known password (it's brute-forced).

    #9 Locky / Zepto
    2016-07-22 https://virustotal.com/en/file/148a...bd1e635cce4425c853e250b21e3d139e0f8/analysis/
    The report mentions a different sample, 3f5ff5d9d0615cc04e644297dcbfa999f6d6930850848f038464d0a486e6b8d0, from 2016-06-27, but its C&C appears to be offline now. This sample is still intercepted by Process Protection (Hollow Process) of HitmanPro.Alert 3.5.
    This newer sample is also intercepted by Process Protection (Hollow Process) of HitmanPro.Alert 3.5. Even with Process Protection disabled, blocked by CryptoGuard. Encrypted files are fully restored. Not a single file lost.

    #10 Bart 2016-06-24 https://virustotal.com/en/file/51ff...e77cb5f26f5ec48d1be42669f368b1f5705/analysis/
    HitmanPro.Alert 3.5 build 545 (or lower) does not offer protection against this threat. Bart ransomware stores your files into 1 password protected ZIP file and deletes the original files. In effect, it's only deleting your files on the disk and is therefore not picked up by CryptoGuard, which watches for crypto-ransomware that encrypts your data on the disk. The data can be restored using a free decrypter from several vendors, using the static password (which is in the txt script) or a free disk recovery tool like Recuva.
    Note: HitmanPro.Alert 3.x blocks the RockLoader with Process Protection (Caller Check). The RockLoader is responsible for introducing the Bart ransomware on the machine. In the real-world, this attack does not succeed. Not a single file lost.

    Conslusion
    So in the real-world, HitmanPro.Alert 3.5 build 546 is only ineffective against ACCDFISA from 2012 (4 years ago). HitmanPro.Alert is signature-less, cloud-less and does not use a sandbox or AV engine to protect against prevalent crypto-ransomware.

    Download
    If you’d like to trial our HitmanPro.Alert 3.5, it is available for download: https://dl.surfright.nl/hmpalert3.exe
     
    Last edited: Jul 23, 2016
  16. guest

    guest Guest

    Thanks for the details :thumb:
     
  17. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,591
    Location:
    U.S.A.
    I use the same technique in my testing. I can't speak for every AV product but it is enough to trigger a reputation failure in Emsisoft AM and Eset which result in detail scanning and monitoring of the process by both products.
     
  18. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,591
    Location:
    U.S.A.
    Not to rally to Zemana's defense since I do not use their products but to set the record straight, MRG did test ZAM in their Q1 2016 360 Assessment comparative; the report link for same I posted in reply #7. It used 70 ransomware samples. The report doesn't elaborate on the type of samples used although it implies 0-day's were used in the testing. ZAM did score 100% in the ransomware testing along with HitmanPro, Kaspersky, Norton, and Webroot. Windows Defender scored 95% higher than Vipre, Avira, Eset, AVG, and McAfee; go figure? The lowest scoring product was Panda at 83%.

    Footnote: MRG is an AMTSO member and as such is using their malware database for samples in their 360 Assessment comparatives.

    -EDIT- The question is why MRG didn't use those same previous 70 ransomware samples for this Zemana sponsored generic anti-ransomware test?

    I am also waiting for tests against javacript based ransonware e.g. Ransom32, RAA, etc..
     
    Last edited: Jul 23, 2016
  19. Rasheed187

    Rasheed187 Registered Member

    Joined:
    Jul 10, 2004
    Posts:
    17,546
    Location:
    The Netherlands
    This was also my first reaction to this test, I'm surprised that no one has mentioned this yet. I mean, it sounds a bit ridiculous to test ZAM against other products that are not using signatures. Also, I believe that CryptoPrevent and Bitdefender are not using behavior based monitoring at all. I expected better from MRG.

    A more logical test would be to test HMPA against MBARW and WinAntiRansom. ZAM should have been tested against other AV's, either cloud based or not. And I haven't read your complete reply yet, but I was also a bit surprised that HMPA didn't perform so well. But I guess you already cleared this up.
     
  20. aigle

    aigle Registered Member

    Joined:
    Dec 14, 2005
    Posts:
    11,164
    Location:
    UK / Pakistan
    Don't forget that if a zero day ransomware hits, all of these including ZAM might be useless. All of them failed to detect the simulator. Very interesting. Very poor performance indeed. I am sure the results of any Antivirus suite will not be different than this.

    If I got any time in recent future I might collect some of these samples and test them against few sandboxes and HIPS( sandboxie, GeSWall and Comodo defence plus) . That will be interesting.
     
    Last edited: Jul 23, 2016
  21. aigle

    aigle Registered Member

    Joined:
    Dec 14, 2005
    Posts:
    11,164
    Location:
    UK / Pakistan
  22. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,591
    Location:
    U.S.A.
  23. aigle

    aigle Registered Member

    Joined:
    Dec 14, 2005
    Posts:
    11,164
    Location:
    UK / Pakistan
    May be. It is interesting. :)
    But it has a high detection rate on VT. How ZAM can miss it?​
     
  24. itman

    itman Registered Member

    Joined:
    Jun 22, 2010
    Posts:
    8,591
    Location:
    U.S.A.
  25. markloman

    markloman Developer

    Joined:
    Jan 25, 2005
    Posts:
    581
    Location:
    Hengelo
    Last edited: Jul 23, 2016
Thread Status:
Not open for further replies.
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.