RanSim Ransomware Simulator test and discussion thread

I'm using avast premier and had 10/10 detected and blocked both with the ransim exe's mentioned excluded and not excluded.

 

It's new, it was created only thirteen hours ago.

Thanks!
So it looks like Avast is doing very well against ransomware. I wouldn't have expected.
Or there is something buggy in the RanSim - Avast interaction that causes incorrect results, but I can't think of what that could be. So I guess Avast is much better at stopping ransomware than I would ever have thought.

 

I tried avast with cybercapture and hardened mode "moderate" disabled and get 9/10

 

The simulator doesn't work here. Nothing happens when I try to launch the exe file. I can see it in Process Explorer, but it doesn't do anything. I disabled KIS but this didn't help, either. Hm...

 

Of note are comments by Eset(Marcos) on Smart Security's failure to detect any of the simulated tests:

They test behavior blocking without distinguishing between malicious and benign applications.

Needless to say that there are many ways how the encryption works so the simulator may theoretically help malware authors to avoid techniques used by the simulator.

In a nutshell, programs that pass the simulator tests may be more prone to encryption by ransomware than ESET.

Ref.: https://forum.eset.com/topic/10556-just-ran-ransim-detection-failed/#comment-53822
​

So security products with strong behavioral blocking capability coupled with cloud rep scanning(assumption that the .exe's of the simulator rate unknown) will do well in the tests. But what is being detected is in actuality, an unknown application behaving suspiciously and not in reality, ransomware encrypting files.

 

FYI - If you want evaluate anti-ransomware effectiveness, stick with the AV lab tests of same. The AV-Comparatives test used 1000 ransomware samples:

https://avlab.pl/sites/default/files/68files/ENG_2016_ransomware.pdf

http://www.av-comparatives.org/wp-content/uploads/2016/11/avc_sp_pcpitstop_2016_en.pdf[plain]

 

It is true, that a simulator is not like the real world.
But it's also true, that the simulator does mass encryption of files.

The encryption executables seem not to be in any malware database, so signature based AVs fail, even when doing cloud scan.

My conclusion is, that software that does not stop the encryption in the first place, is vulnerable to daily fresh sample of ransomware.

 

I can confirm the bug is still present in RanSim 1.0.2.2.

People can perform a file compare to verify the RanSim bug using the fc application (part of Windows) using the following steps:

1. Install HMPA (make sure the license icon is green, bottom right of the GUI)
2. Start RanSim
3. Open a command prompt and type the following command (make sure to replace John with your own profile name):

fc C:\Users\John\Documents\RanSim\TestDirectory\TestFiles\*.* C:\Users\John\Documents\RanSim\TestDirectory\Scenarios\InsideCryptor-TestFiles\*.*

Spoiler: Output

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\c10.png and .\c10.png
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\c11.png and .\c11.png
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\c12.png and .\c12.png
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\img10.jpg and .\img10.jpg
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\img11.jpg and .\img11.jpg
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\img12.jpg and .\img12.jpg
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\img20.jpg and .\img20.jpg
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\img21.jpg and .\img21.jpg
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\img22.jpg and .\img22.jpg
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\img30.jpg and .\img30.jpg
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\img31.jpg and .\img31.jpg
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\img32.jpg and .\img32.jpg
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\MOCK_DATA1 .csv and .\MOCK_DATA1 .csv
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\MOCK_DATA1.docx and .\MOCK_DATA1.docx
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\MOCK_DATA1.pdf and .\MOCK_DATA1.pdf
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\MOCK_DATA1.pptx and .\MOCK_DATA1.pptx
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\MOCK_DATA1.xlsx and .\MOCK_DATA1.xlsx
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\MOCK_DATA2.csv and .\MOCK_DATA2.csv
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\MOCK_DATA2.docx and .\MOCK_DATA2.docx
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\MOCK_DATA2.pdf and .\MOCK_DATA2.pdf
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\MOCK_DATA2.pptx and .\MOCK_DATA2.pptx
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\MOCK_DATA3.csv and .\MOCK_DATA3.csv
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\MOCK_DATA3.docx and .\MOCK_DATA3.docx
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\MOCK_DATA3.pdf and .\MOCK_DATA3.pdf
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\MOCK_DATA3.pptx and .\MOCK_DATA3.pptx
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\MOCK_DATA3.xlsx and .\MOCK_DATA3.xlsx
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\test1.docx and .\test1.docx
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\test2.docx and .\test2.docx
FC: no differences encountered

Comparing files C:\USERS\JOHN\DOCUMENTS\RANSIM\TESTDIRECTORY\TESTFILES\test3.docx and .\test3.docx
FC: no differences encountered

 

This is absolutely true and the sole reason why so many home users and businesses are infected with ransomware, despite the presence of up-to-date AV solutions.

And this problem is not new. Ransomware just shows itself to the user (for payment) so that it becomes evident you are infected. Prior to ransomware, computers were infected just as often, you just were not made aware by the malware.

Let that sink in ...

 

@itman
Do you don't think that such explanation on Eset forum is quite funny?...something like "it's not a bug...it's a feature"? This test was not made to test Eset but to test every apps/settings and its combination installed on system and working in real-time...so making opinion about test according to Eset's skills looks "a bit weak". No matter how many technics are used by ransomware in real world and how many of it Eset is capable to detect and block - this test checks some of them an by this way try to give answer for users how good/bad could be their way of protection while facing real malware.
BTW - I tested SpyShelter ... this is my daily real-time protection ... and the score is 10/10. The test was made on Vista...yes the test is working on Vista also. The "My Document" folder is on the list of protected areas in SS's options, so it was enough to block the start of every single scenario. Maybe it is not a "sophisticated" method, but SS correctly detects such actions, which are simulator of operations of real malware.

 

I can accomplish the same in Eset by just creating a HIPS rule to monitor write activity to the "My Documents" folder. However, what is being detected is file modification and not actually ransomware encryption activity.

Again what Eset was stating is that this test tool does not perform actually encryption activity against system areas targeted by ransomware. As such, it really is not effective in determining how well a given product will perform against an actual ransomware attack.

Important to note is this. The simulator is creating its own test files and then doing encryption activities against those files. This type of activity in many security products would not be considered malicious since the app is performing the encryption against files it created. This is not the case with actual ransomware behavior; it is encrypting files already in existence and not in any way associated with the simulator app.

-EDIT- Eset SS and NOD32 - ver. 8 actually does have a HIPS file modification option to monitor for "ransomware like activity." Eset added this to ver. 8 since that older ver. doesn't have by default ransomware protection built-in. If this option was applied against the "My Documents" directory for example, it would be monitoring for encryption activity being attempted against files it that folder.

 

My suggestion to the developer of this test tool is to do the following.

For each installation:

1. Generate a random .exe name
2. Modify the hash value of the random .exe to be unique.

At this point, we have a 0-day simulation.

Also prior to installation, require the user to create a test folder within "My Documents" containing files that will be encrypted by the test tool. Require that this folder location be specified at installation time.

Now run the simulator and see how your existing security solution performs against encryption activity in the user specified test folder.

 

Comodo firewall blocks all of them no problem.

 

Thanks, I think those may be good ideas.
I have no idea whether the developer is following this thread, as the thread was started only 41 hours ago.
I think it would be a good idea if you could contact the developer offering your suggestions.

 

Correct, if a tool offers pro-active protection, it should at least alert about this simulator.

What about the rannsomware simulation tool from Sophos, will it be released, and can you compare it to the RanSim tool?

OK thanks for the info.

 

Spiceworks has a thread on it here that was started in mid-Oct.: https://community.spiceworks.com/topic/1880114-ransim-a-ransomware-simulator . Comment that stuck out to me was how well Bitdefender performed. That makes sense since BD has a very aggressive behavioral blocker; on par with Emsisoft's.

Again what a ransomware test needs to demonstrate is effectiveness against actual malicious encryption activity, not behavior that could be associated with ransomware such as encryption activity. Such activity could very well be benign and valid activity. For "security jockeys," the more alerts the better. They have the "smarts" on how to interpret and respond to them. However for the average user the less manual intervention required, the better the security solution.

 

Hi

Anybody tested against SBGuard? CryptoPrevent failed according to the Spiceworks link in post #41

 

Another free tool for testing anti-ransomware defenses by WatchPoint

https://blog.watchpointdata.com/wat...a-free-tool-for-simulating-ransomware-attacks

 

Thanks for interesting comments...you are right...I noticed that simulator installs own folder with files needed to perform such test and probably it can "darken" the results. Below folder created on my disk



I think it's can be due to some possible allowing rules created by security app made during installation - my test was performed after cleaning all rules created while installation so every next detection was "fresh" for SS.

 

Er, well, yes, that information is in the RanSim documentation details, of course.
"will automatically create the test environment (the TestDirectory folder and its contents)"
(Search page: Ctrl+F, keyword: TestDirectory)

 

Hi all, I have tested Emsisoft IS and Malwarebytes 3.0. Emsi exceeded the test with 10/10.
By malwarebytes no allert.

 

Yeah, I knew about this test. A bit too risky for my liking since it requires installation of a root cert. and actually encrypts your existing files. So proceed with caution on this one.

Here is another far less risky ransomware simulator from Microsoft's TechNet that also uses Powershell but does not do any actual file encryption. It also allows you to create and specify the directory that will be used for the test: https://blogs.technet.microsoft.com...mulating-a-ransomware-attack-with-powershell/

 

https://www.youtube.com/watch?v=6KCHjGI6Q6w just finished watching...

 

Norton Security 10/10
MBAM 10/10 and
I can deal with the problem
Regards

 

Actually, Spiceworks had multiple threads going on about RanSim. Below is a comment from the another thread: https://community.spiceworks.com/topic/1874415-ransomware-simulator-we-failed?page=1 that I found interesting:

I just tested it in my lab. Intercept X reported five ransomware detections (out of five tests), and none of the test files ended up encrypted, and all of the tests reported as "unexecuted." The RanSim tool reported this as "0/5 vulnerable" and "0/5 invulnerable." I'm not sure how to interpret that scoring, but it seemed effective to me. Don't take my word for it, though. You can grab a free trial of Intercept X at https://secure2.sophos.com/en-us/products/intercept-x/free-trial.aspx.

Testing note: It may help to disable "Automatically clean up malware" in the Threat Protection section of the policy. Otherwise, the first detection will kick off a cleanup job, which runs in the background while the other tests are trying to proceed.
​

Again the above indicates that RanSim is not accurate at determining anti-ransomware effectiveness.