Small scan test of Hitman pro and Prevx

I got recently from http://malwareresearchgroup.com/ a set of 1100 new pieces of malware. (They give sample sets when requested for us at Gizmo's Freeware for testing purposes.)

Hitman pro found 1040/1100.

Typically 2-3 AV:s recognized each beast. If we ASSUME that all engines are approximately equal, the scanning performance of each single engine is thus 44%.

I scanned also with Prevx alone. It found 40%.

The power of using several scanners is seen.

 

Interesting. I am kind of surprised Prevx did not do better. Hitman Pro is incredible.

 

what were your settings on Prevx, thanks

 

That is a wild assumption that I have massive difficulty believing.

Malware detection should be very closely correlated amongst the various AV engines.

I am also skeptical that Prevx would only detect 40% .... maybe some settings tweaks are required.

EDIT: I take back my comments, I did not realise it was 0-day

 

Medium - low - low

 

It was scanner test, not on-execution test.

 

This is a 0-day set. We would not get 94.5% detection rate if correlation would be high.

 

I think Prevx will jump to 90+ % when you try to execute those malicious files.

Btw, I don't take tests seriously? Why?
When you encounter any malware (one day or another) and your AV/Anti-Malware program detects it, you're good!

If your protection software has 99.99% detection rate and you fail that one malware, you're infected! That is what matters.

 

Also Prevx strengths is shown when you execute the files instead of just scanning the files! But I don't recommend doing so unless you have an Isolated VM!

TH

Edit: LagerX beat me to it

 

Why would you be skeptical. Prevx is a great program and I absolutely love it. But 40% is not extremely bad in my opinion. Especially if you realise Prevx power is not on-demand scanning, but in realtime (two people were earlier ). You should take into consideration that there are several problems with MDL, but in my eyes the most important is that 95% of that "malware" (is it really?) doesn't reach any consumer pc's anyway.

 

Also Prevx users with a license have a guarantee of free clean up with a remote session if necessary!

TH

 

The only point I wanted to make here was the fact (that everybody here should know anyway): the detection rate of 0-day malware of AV:s is too low. In addition to them one definitely should use sandboxing/LUA.

 

For me I would have to say no for the average user! But for ones that like to go find malware then yes! IMO

TH

 

I take back my comments, I did not realise it was 0-day

 

LUA? Only for those who "like to go find malware"? Really??

 

I'm sorry but I'm not LUA or UAC user and none in my family either and they never get infected as they are average users and some have NOD32 and Prevx and my Mother Inlaw uses MSE all in admin accounts and there's no problems! Also I'm not a advocate of LUA or UAC but like I said it's IMO!

TH

 

Oh snap

 

Could you break down this math again as I don't understand what you are trying to say. I see where it says HMP detected 1040/1100. Where are you coming up with the 44% assumption?

 

He is breaking down the number of scanners and the possibility of each finding malware. More is better. I like his test and I also know he is doing a pretty good job.

He has tested a couple of products for me against the test bed and I would only post if they were positive, so dont ask who they are.

TH, question about Prevx. If on execution is when really it detects, why does it have a scanner then. I am just trying to understand. I see where people write that on scanning it wont detect, but if something opens it gets it. Why the scanner?

 

Just noticed this thread and want to make a few points.

With zero day / very early life samples, most AMs will detect between 40 – 60%, depending on age and type of malware.

A good proportion of zero day / very early life malware never seem to make it in to the wild in any significant way, so testers who use sites like MDL are not using zero day / early life samples.

A detection rate of 40% on very young samples is fine. We test Prevx regularly on various samples and its as good as the top five.

One thing to consider is that ALL AMs will miss about half of zero day – very early life samples, however, Prevx will block the action of a good proportion of malware that runs on a system via its SafeOnline technology.

I know this test was just an on demand test and does not purport to be anything more than this, but, it serves as stimulus. Consider the two main ways security apps are generally tested. 1) Static 2) Dynamic. In our view, dynamic testing, using a realistic vector represents the minimum specification for any meaningful test.

There will be more and more testing organizations out there which will perform dynamic tests, but you need to look beyond this to really assess the efficacy of applications. For example and with Prevx in mind, you could execute a zero day financial malware sample and this may go undetected by Prevx. Most testers may then conclude Prevx has failed. This is not necessarily so. You need to be able to establish if the sample, once executed has been able to operate and achieve its objective.

You need to be able to assess if the security application actually prevents the action of executed malware, not just if it stops it executing. Doing this introduces a new level of complexity in to testing and requires the use of custom designed simulators or ethically controlled real botnets etc.

Given it is a fact that about half of all zero day samples will go undetected, vendors need to start looking at technology like SOL which silently and automatically isolates the browser and other applications from the effect of infection OR some whitelist technology to prevent all unknown code from executing.

Regards,
Sveta

 

Because when you open the EXE file then Prevx can read the files during extraction and also try to download other files as most nasties try to do! In my own testing that is what I see as not all files in the EXE are malicious. But Joe or Marco would be a better person to ask such a Question to!

TH

 

(1-p(detected))^5*1100=1100-1040

 

yeah, but they are eating dinner.

 

It does not hurt. It may add some extra protection. And it makes detection quicker/light. (Moreover the average Joe wants to scan to feel safe.)

 

Maybe I am missing something as I never claimed to be the brightest in the world. And I've never been accused of being up to date. But is it en vogue to guesstimate detection rates regarding such a small sample size? Shouldn't there be some way to find out how many detections each of these antimalware scanners/engines made instead of guessing? I realize there is a total number of samples detected. But isn't it possible one engine detected a lot more than some other? And some other engine detected a lot less than the average? I think Ikarus usually detects a lot more than Dr. Web and I'm not sure if Emsi is just the antispyware scanning or the Ikarus av and Emsi antispyware. But if there was one engine that usually detected a lot (Engine A) and another engine that did not (Engine B), wouldn't it be more likely that all the detections of B were included in A?

I think the sample size would skew things somewhat but when it is small like this it would be better to get actual numbers of each scanner's detection- including maybe each of the A&B engines of Gdata.