VoodooShield/Cyberlock

I have had a couple of people ask how well the new Comodo 10 beta performs with the pre-execution blocking efficacy test, so I tested it with the first 1,000 samples, and it did really well... 93.1%!

www.voodooshield.com/artwork/ComodoTest.png

There is no point in making a video on this test, but I have the original .webm video if anyone would like to see it.

Also, there is no point in testing with the 3,000 samples, since the results are going to be about the same either way... due to the magic of random sampling.

 

Yeah, it does, thank you! Yeah, I really like the method that you guys and Vlad came up with for dealing with dismhost.exe (checking the hash of the dismhost files in the Windows folder), but there is a little bug in there somewhere. It might be actually a permission issue in obtaining the hash of the dismhost in the windows folder, if it is a protected file... something like that . Hopefully Vlad can look at this soon too.

 

What settings is in this CIS test (default or...)?
I tested also but it didn't do so good.
I would like to see that video.

 

100% default settings... except I did not install Geek Buddy during installation.

Assuming you are talking about testing with EfficacyTest.exe... I must have the magic touch or something, because it seriously works perfectly for me every time. And actually, that was another reason I wanted to run one more test... I wanted to see if I forgot to tell you guys anything while running the pre-execution efficacy test with EfficacyTest.exe.

It really is as simple as this:

1. Create a VM (I use VirtualBox)
2. Copy the malware samples to the VM... randomize the samples to reduce the sample set size and to ensure you are only using random samples.
3. Install and update the security software you are testing.
4. Run EfficacyTest.exe and execute the samples.

EfficacyTest.exe was never meant to be released to the public, so it is not that well written, but it worked perfectly for my tests, so I thought it would make it easier when other people wanted to perform similar tests. And we can build it out a little more, I just do not have the time right now.

Besides, I would actually prefer that testers use different methods while testing, and not use EfficacyTest.exe... quite simply because we already know the results if they follow my procedures . Although I certainly do not mind anyone verifying my tests. The whole idea of the VoodooShield Challenge was for different participants to create their own testing methodologies, and to make these tests truly their own... that way we can see different points of view, especially since no test is perfect. Also, I thought the VoodooShield Challenge would be really fun and insightful for everyone, and hopefully inspire everyone to start testing on their own, that way when they discuss malware on MT or wilders, they have actual numbers to confirm what they are saying.

Also, the test I performed was a pre-execution efficacy test... which IN MY OPINION, is one of the most important tests, simply because if you can stop malware pre-execution effectively, you are MUCH BETTER off... as opposed to rolling the dice with behavior blocking, sandboxes, etc.

Think of it this way... if VoodooAi determines that a file is unsafe (> 0.9000 or so), why even run it in a sandbox? The odds are extremely high that there is something wrong with that file, so it is in the user's best interest to not even run the file at all. If the file is determined to be Suspicious by VoodooAi, then sure, run it in a sandbox. If the file is squeaky clean with the blacklist and VoodooAi, auto allow the sucker .

But my point is this... I would like to see the results of other tests that the participants perform for the VoodooShield Challenge, that does not focus on pre-execution.

Sorry for rambling... I just realized that there might be some other questions, so I figured I would answer them before they were asked . Thank you!

 

Sorry for off....

Run 36 malware with EfficacyTest, CIS on default settings only antivirus settings is "On Access".
Results:

 

Yeah, that is what I thought you might be referring too... I saw that post, but could not remember who it was that ran the test, and forgot where to find it.

I have no idea why it is not working for you. Did your procedure differ from what I listed above?

 

I think not.
Everything is the same, Autosandbox didn't show at all...
I will try now again (i have snapshot with CIS just installed)

 

Cool... you might look at my videos on youtube, or I can upload the Comodo .webm video so you can see if there is possibly a difference between our procedures.

 

I watch all video you add to youtube and would like to see that CIS video.

BTW: here is new test with the same 36 malware...

 

ok this might have been coverd already in this thread but I was wondering how VS handles torrent clients?

thanks

 

Blocked, yes. Sandboxed, no. Here's a pretty picture.

 

This version has currently froze just after allowing dismhost.exe as above. I'll send logs.

 

Cool, I will upload the file somewhere... I am going to stop uploading big videos to voodooshield.com because it is making our backup go over the limit, but I will find a place.

I am getting ready to test the samples you sent me... BTW, did you ensure that all of the files were actually executable before running the tests?

 

Yeah, what SHvFl said (thank you SHvFl), but I would like to add that you can add them as a web app if you like, so that VS toggles with them in Smart Mode.

 

Hmmm, that might be one of the issues... that version of dismhost.exe is not known to the blacklist. Although, it should not matter because VS is supposed to check the hash of the current dismhost files on the computer. Thank you!

 

VS thanks I was wondering why they did not turn VS blue .. because they really are a web app.

" VS is supposed to check the hash of the current dismhost files on the computer. Thank you!"

and all my blocks had the same hash but not same as posted above.

 

Cool, thank you... I will look at them. The last version handled searchprotocolhost.exe a little different to test... but dismhost might have been what is causing the freeze issue all along. I think the code for the new method of handling dismhost was added right around that time (before 3.09). You might be on to something .

Since you have had more problems with the freeze issue than anyone, wouldn't that be rewarding if you cracked the code and figured out the freeze issue?

 

Maybe Dan, but remember VS has frozen regularly after prompting for Software Reporter Tool and some Command Lines as well.

 

Yes, all files are .exe, maybe 2 have .scr.
I tested the same malware folder vs VoodooShield and all get block.

 

Yeah, we will make a running list of the suspects to give to Vlad .

searchprotocolhost
dllhost
dismhost
softwarereportertool (google chrome)

 

I just test 19 clean files with EfficacyTest and VS.
VS block 2/19
Is this because of .msi?

******************** Efficacy test started on 16.7.2016. 22:24:47 ********************
Allowed!! C:\Users\Av-Gurus\Desktop\CLEAN\360TS_Setup.exe
Allowed!! C:\Users\Av-Gurus\Desktop\CLEAN\Advanced_Uninstaller_Free_10.exe
Blocked: C:\Users\Av-Gurus\Desktop\CLEAN\crystal_security_3.5.0.184_setup.msi
Allowed!! C:\Users\Av-Gurus\Desktop\CLEAN\diskboss_setup_v7.0.32.exe
Allowed!! C:\Users\Av-Gurus\Desktop\CLEAN\folder-lock.exe
Allowed!! C:\Users\Av-Gurus\Desktop\CLEAN\foobar2000_v1.3.10.exe
Allowed!! C:\Users\Av-Gurus\Desktop\CLEAN\FreeFileSync_8.3_Windows_Setup.exe
Allowed!! C:\Users\Av-Gurus\Desktop\CLEAN\FreemakeVideoConverterFull.exe
Allowed!! C:\Users\Av-Gurus\Desktop\CLEAN\gu5setup.exe
Blocked: C:\Users\Av-Gurus\Desktop\CLEAN\install_virtualdj_pc_v8.2.3291.msi
Allowed!! C:\Users\Av-Gurus\Desktop\CLEAN\Opera_beta_39.0.2256.30_Setup.exe
Allowed!! C:\Users\Av-Gurus\Desktop\CLEAN\PotPlayerSetup64.exe
Allowed!! C:\Users\Av-Gurus\Desktop\CLEAN\privazer_free.exe
Allowed!! C:\Users\Av-Gurus\Desktop\CLEAN\rufus-2.9.exe
Allowed!! C:\Users\Av-Gurus\Desktop\CLEAN\setup_x64.exe
Allowed!! C:\Users\Av-Gurus\Desktop\CLEAN\SmartyUninstaller4.exe
Allowed!! C:\Users\Av-Gurus\Desktop\CLEAN\tagscan-6.0.14-setup.exe
Allowed!! C:\Users\Av-Gurus\Desktop\CLEAN\WDCFree.exe
Allowed!! C:\Users\Av-Gurus\Desktop\CLEAN\zp1210free.exe

Files Missed: (17 / 19)
Efficacy: 10,5%

 

@VoodooShield ,
Dan, I just found this in the DeveloperServiceLog.

Code:

[07-17-2016 05:42:14] [ERROR] - HandleSingleProcess(-2145452000)::Failed to reply to driver (0x801F0020)

I don't know if that helps or not.

Thanks.

 

Ooops... that is because VoodooAi does not currently analyze .msi files. What happens if you enable the blacklist scan and rerun the test?

 

Very, very cool, thank you!!! That really helps a lot... was that right around the time that VS froze?

 

Yep! I can't say if it was exactly the same time but from the time I posted about the freeze it is similar.