Prevx1 is far behind other HIPS in a HIPS test

There is a good HIPS test done by the reputable website called AV Comparatives.

9 HIPS* products were tested:

BufferZone Home Pro http://www.trustware.com
CyberHawk 1.2 http://www.novatix.com
DefenseWall HIPS 1.7.1 http://www.softsphere.com
GeSWall PE 2.5 http://www.gentlesecurity.com
Kaspersky Internet Security 6.0 http://www.kaspersky.com
PrevX 1 2.0 http://www.prevx.com
Safe'n'Sec Personal 2.5 http://www.safensoft.com
Sandboxie 2.62 http://www.sandboxie.com
ViGUARD Platinium 12 http://www.viguard.com​

People can read the report on its website. Follow this step:
1) Go to http://www.av-comparatives.org/ *
2) Search for "Comparative of various protection tools"
3) you will see this line:

Comparative of various protection tools　　　October 2006　　Report (PDF)​

　Click on "Report (PDF)"

=========================================================
* PS:
- It's okay to post this link since the reader can still read the the important notes contained in this page. Checked.
- HIPS = Host(-based) Intrusion Prevention System

 

The tested program is configured and tested properly, and the result has been confirmed by the vendors.
Here's a brief description of the result.
All pass (perfect):

BufferZone Home Pro (sandbox HIPS)
DefenseWall HIPS 1.7.1 (sandbox HIPS)
GeSWall PE 2.5 (access control HIPS)
Kaspersky Internet Security 6.0 - PDM (behaviour blocker)
Safe'n'Sec Personal 2.5 (behaviour blocker)
Sandboxie 2.62 (sandbox HIPS)​

Partial Pass:

ViGUARD Platinium 12 (behaviour blocker) -- all pass only in interactive mode
(The tester found a serious bug in its automatic mode. This bug made ViGuard not able to find 5 samples of malware. However it would not be all pass if it is in the interactive mode)​

Fail:

CyberHawk 1.2 (behaviour blocker) -- 1 fail
PrevX 1 2.0 (behaviour blocker) -- 5 fails​

Note:
- TruPrevent is not included as Panda does no longer wish to take part in our tests. Too bad that Panda does not like us to see its performance.
- all participating vendors have fixed the problems after the test result is published.

 

Some comments on Prevx1 performance
*Readers feel free to (mis)interpret the results/comments on its performance. Please do your own diligence!*

As the tester said, the 5 samples were missed because Prevx:
a) did not have them in their database and therefore did not recognize them as malicious files and
b) because Prevx did not block the samples (the actions done by the samples, so, neither by their behavior)

The test is done when the tester is online. It is very surprising to see that Prevx1 can grossly miss *5* samples out of 40 (12.5%) which the majority of HIPS can manage to miss none of them.

Prevx1 is proud of its unique automated malware research. It claims that Prevx1 can see more threats and protect you earlier because other security solutions have to hunt and catch each new malicious threat before they can create a fix. It has a community database which Prevx1 users can report the mew malware as soon as the malware reaches any one of the user computer. This let them see and protect you faster than other security programs.

How can it close the gaps between 0day/new malware attack and signature update by AV/AS etc. when it can grossly miss 5 samples in this test? It is doubtful how effective its community database is.

Prevx1 failed completely in Greenborder test
Here's another test about Prevx1 at https://www.wilderssecurity.com/showthread.php?t=150840
This is a test from Greenborder. There are currently five checks which the test performs:

1. Attempts to steal confidential files from My Documents (Javascript copy of files to a desktop folder)
2. Simulates installing a keylogger (writes a blank space to the registry run key)
3. Searches files for 'pass' (searches inside confidential files for text)
4. Attempts to reveal passwords from protected storage (currently disabled since I need to hide parts of the password)
5. Attempts to open disk manager via System Call (Could be any system call with parameters like 'delete volume')

I tried to run that test on various products. Here's the test result:
McAfee On-access -- warn a suspicious script is running. It also stops the script from calling a file. All pass.
Sandboxie -- the script is trapped into the sandbox. All pass.
Prevx1 -- total failure. It doesn't do anything - no blockage, no popup. All fail.
Firefox/Seamonkey/Opera -- passed as expected. Only text script is shown. Nothing can be executed. You can see how bad IE is.

Prevx1 on-demand scan seems to perform worse than AV
I have done several small tests for my personal reference. As far as on-demand scan is concerned, Prevx1 is proved to be very weak. Prevx1 managed to detect 3-5 malware only while Kaspersky (KAV) / Avira AntiVir can manage to detect 15-20 malware on average. It is done online.

By the way this somewhat confirms what one report says. I once read a report which compare on-demand scans of many anti-malware, including Prevx1, but I have missed that link. The result is also astonishing. If my memory serves, Prevx1 managed to detect 30-50 samples only while the top AVs can detect 300-400 samples.

Although I'm told from the vendor that that test isn't run properly (however they couldn't provide that link), this makes me wonder how effective Prevx1 is really anyway (the difference is too large). If anyone come across this link, please tell me.

It is worth noting that you may not use demo/test tools to test Prevx1. It is because Prevx1 may simply add the tools into their database (ie blacklist them), so they block it without any problem. Anyway, to make sure it is not blocked by blacklist, it is the best to disconnect from the Internet when you do such tests like termination, buffer overflow, leaktests and so on.
In this regard, we can test how good it is to defend against these attacks by its internal protection capability (not by simply a blacklist) unless the tool is also blocked by its local database!
(Note: sukarof reported that "Prevx1 do not add any leaktests or malware testing tools to their database". I'm not sure which party is right. But I once run the GhostSecurity Registry Test [about 1 month ago]. Prevx1 alerted as soon as I double clicked it)

Conclusion
Although Prevx1 is quite different in that it works like an automated-HIPS which will answer the prompts on behalf of the user (which is nice to the noobies or people who don't bother or can't answer the prompts properly), don't fall into traps you are very safe with Prevx1 simply because it takes care of them all, let alone use it as the only security product (it is what the website has stated too - Prevx1 can be used to replace your current security suite).

Although neither test can be used definitively to prove its performance/efficiency, it should show some alerts on how well it may perform.

 

Hi!
This has already been discussed at lenght recently
You may want to read in here:

https://www.wilderssecurity.com/showthread.php?t=152694

And at castlecops:

http://www.castlecops.com/t169603-Prevx_and_Green_Border_tests.html

Cheers,
Fax
P.S. But very nice summary

 

Prevx1 needs time to get polished. The idea is good and makes HIPS popular amongst the majority of the users.
In one of my posts, I recommended the Prevx1-Team to test Prevx1 constantly in the swamps of the real internet.
Notok confirmed that this is done in practice and I hope it's true.

Prevx1 might be weaker at this moment compared with other HIPS, but it has the POSSIBILITY to become a very strong all-in-one security software and there is nothing better than prevention.
The beauty of failures is that they can be fixed and those 5 failures are already fixed by Prevx1.
That doesn't mean that Prevx1 is inferior, all security softwares fail, that's what they have in common.

The only reason for me to give up on Prevx1 is, when the Prevx1-team isn't motivated anymore to fix problems.

 

Just to clarify: Prevx1 do not add any leaktests or malware testing tools to their database. They have several times explained why. It is everyones choice to accept that explanation or not.

 

right. and it was able to protect against 35 otehr various samples, which should not be considered worthless. I guess prevx like other hips would be able to block hundreds of malware which are not detected on demand by most other antiviruses.
even if some programs need still some improvements, they still offer additional security.

 

IBK,
What mode was Prevx1 in when you performed these tests? If it was mentioned in the pdf I missed it.
Would it have performed better in expert mode since it would then rely more on its actual HIPS functionality and less on the community? Of course, that would defeat the user-friendliness that Prevx1 is known for.
Thanks

 

Hi Wai Wai,

Thanks as always for you’re enlighten comments on Prevx1.

As several posters have mentioned if users want a fully blown HIPS product to query everything running we can provide this. Prevx1 Home and Pro had this functionality but it was clear from the average user this wasn’t what they wanted.

As we know from many forum entries tests are very subjective. Prevx runs internal tests on around 10,000 samples per week of malware received. It's surprising how many of the top AV/Anti-Spyware products fail to detect the same samples.

Prevx1 malware sample set is always the most current given it is supplied directly by users finding Prevx1 the only security product able to cleanup infections. Our growing user base is testament to the number of users who have malware removed (where other products have failed) and have no issues with performance.

As pointed out in many forum entries no security product is perfect and your conclusions can be equally applied to many security products.

You’ll be pleased to hear we have included in the next release a popup dialogue for the Greenborder test.

Regards,

Prevx Support

 

Interesting test, but probably not fair. HIPS like Kaspersky's PDM are of little value to an average user who is not interested in and does not know how answer an endless series of popups, mostly from programs that are not malware. I have tried PDM and many or most of the programs I installed from trusted sources caused various cryptic popups warning of things like writing to the registry or injecting into other processes. Presumably, such programs would have been falsely classified as malware by HIPS like PDM if using the methodology of the above test. That is, the above test does not measue how many false positives the HIPS find.

Regarding the results for Prevx1, after looking through the earlier discussions, I note that the user was asked to allow an unknown program to run. I find this little different from conventional HIPS giving an cryptic warning. Also, since these 5 programs were not marked as "Good", they may well later have been marked as malware automatically by Prevx1, although the first user would have been infected before this happened.

 

no, not automatically.

@se7engreen: was asked often, and as I said, I did not use Prevx in abc mode, use the pro mode and tried of course also with the higher settings in expert mode.

 

All pass (perfect):

Safe'n'Sec Personal 2.5 (behaviour blocker)

Wow! Safe'n'Sec Personal 2.5 is in its first beta stage and it passed all test. I better go try that beta out .

dja2k

 

i was gonna try safe n sec soon lol.
lodore

 

Hmm... I wonder where sukarof got that quote (he didn't say)? In any event, it is my opinion that one of the single worst things that can be said of any security program is "It can't be tested." There never was a horse that couldn't be rode. (There never was a rider that couldn't be throwed.)

Prevx's database CAN be tested, as was demonstrated by AV-C. The only challenge is that you can never test Prevx TWICE using the same testbed. Well, you *can* do so, but it would be pointless because Prevx ingests new threats into its db as fast as they come. Gobble gobble gobble. Chomp. Burp!

Test again, Prevx will win!!! But I still prefer SSM and Cyberhawk and Popeye's Cajun-style Fried Chicken.

 

If the idea is meant to be "reducing prompts / help users to make decisions", this is the new HIPS which many are doing: Online Armour, Cyperhawk, Safe 'n' Sec etc. It is a tendancy that more and more HIPS will implement ways to help average users to answer the prompts.

If the idea is meant to be its "community defence" or "automated malware research", it appears only Online Armour does that too. However this nice implementation tends to be hype. It is not as effective as I supposed. The community defence is not like a magic. It is still lagging behind others. There are still misses every day. It may also have occassional mistakes/errors on the database.

Think twice, here may be the weaknesses of "community database / automated malware research" (rough ideas):
- how many users are in the community database - this affects the effectiveness?
- how well does it handle so much info coming from many users each day?
- when a malware reaches the user, how do you make sure you receive the correct copy (sometimes the file might be corrupt during the transfer)?
- how smart your "automated malware research"(AMR) is? How does it actually work?
- when well does the so-called "automated malware research" perform? It is still doubtful without knowing how approximately it works?
- how accurate does the AMR is? What if it makes a mistakes? How can you correct it?
- do you have enough secuirty staff to verify the results generated from AMR?
- can you afford the workload and update in time?
- what if the malware is sneaky and doesn't do malicious activities instantly? Some may wait for a specific period before it triggers itself, or it only trigger on a specific date/time? On normal days, it hide itself very deeply? How can you identify this kind of malware?
- what if the user who has allowed the malware to run before you have updated the sample to the database?

OK. That's brainstorming.
I know the vendor is not going to answer it since it is business secrets, but I just ask anyway.

That's we all like to see.

So do other participants, ie Viguard & CyberHawk, which is good news.
As far as I know, Viguard is the fastest to fix the fail in this test.

Yes, all security softwares (will) fail.
The more you fail, more chances that you are worse than others.
The less you fail, more chances that you are better than others.

 

I'd like to know in what way Prevx1 failed against the 5 it shows. Did Prevx1 alert to the launching of the 5 samples and the tester clicked 'Allow' to see if Prevx1 could block the running of these samples through 'other' means. Is it that the 5 samples ran without Prevx1 alerting in any way whatsoever. Just want clarification.

muf

 

I have added responses to sukarof quotes now:

(Note: sukarof reported that "Prevx1 do not add any leaktests or malware testing tools to their database". I'm not sure which party is right. But I once run the GhostSecurity Registry Test [about 1 month ago]. Prevx1 alerted as soon as I double clicked it)​

Yes, it is supposed so.

But think twice, it may not be true.

Who knows how effective or flawless the automated malware process is?
It may be that the sample is stuck in the database, and no one remember to review it, or the automated malware process goes wrong and the sample somewhat can't pass to the researcher. All sorts of possibilities.

We need to look into its community database and its process in order to make sure it works as it intends to.

I once thought if I uploaded the malware sample to the multiple-scanner website (ie VirusTotal or Jotti), most, if not all, participating anti-virus company should be able to update their signatures. However I re-upload the same malware to the multiple-scanner website after a few month passed, it is surprised that many still miss that malware.

So don't think so simply. It is always easier said than done. There are many practical and minor problems (which we may overlook) that may obstruct the achievements of this goal.

There is a report from AVComparatives which describes the progresses made by anti-virus companies since alst comparative. OK, every anti-virus company had recieved the missed samples? Can they manage to detect all the missed samples after 3, 4, 5, 6 months? Do you know what's the result?

Go to http://www.av-comparatives.org/
The reports are under:
7. On-demand comparative (August 2005)
9. On-demand comparative (February 2006)
11. On-demand comparative (August 2006)

Find out the answer yourself.

(Note: Interesting read! Good job, AV Comparative and the author of the website. )

It appears Prevx1 community defence or automated malware research is nothing like a magic solution, so it is not significantly different than other products. HIPS Choices are open now:
- Prevx1
- Online Armour
- Cyberhawk
- Safe 'n' Sec
- GesWall (an interesting HIPS )
- don't forget sandbox/virtualization HIPS

 

Count me in too.

Safe’n’Sec has an intelligent decision maker. How good is it to help us to make decisions on prompts?

Does anyone know what it is?

This is the quote from http://www.safensoft.com/security.phtml?c=83&id=1100:

Safe’n’Sec Personal is 98% efficient in combating malware of various nature according to AV Comparatives test!

In August 2006 AV Comparatives laboratory by Andreas Clementi has conducted a test of 15 classical antivirus engines. A bunch of antivirus solutions has been tested on 551.795 malware samples chosen at random. The test has shown the middle index of classical anti-viruses efficiency in combating various types of malware, viruses, backdoors, worms, Trojans etc., is approx. 509.000 malware samples blocked and approx. 42700 samples missed
http://www.av-comparatives.org/

Then the testers randomly picked 60 various malware samples from those 42700 samples missed in order to check the efficiency of Safe’n’Sec behavior analyzer. The result was the following: 59 malware blocked and 1 missed which means Safe’n’Sec technology has shown a 98% efficiency in combating malware of various nature.
​

I can't see such kind of test in AV Comparatives
Where is it?
[Answered by IBK, the author of AV Comparatives:
Some tests are so small that there is simply not enough to post it. in such cases i usually post only a link to the press release (which in this case i have not yet done)]

 

Good thing I am a beta tester for Safe'n'Sec, so maybe I should try it out again. I already tried it out, but wasn't really too interested in continuing its testing, now I am.

dja2k

 

IBK, it would be great if you could test how well HIPS can catch the missed samples.

 

Quote is to long to quote and is above:
Wai Wai,
after reading the main site I like the fact it only uses 5mb ram and 2percent cpu wish is amazing! and 20mb harddrive space. it sounds like my kind of app.
I would like to read the av-comparatives article as well
lodore

 

some tests are so small that there is simply not enough to post it. in such cases i usually post only a link to the press release (which in this case i have not yet done).
the resource usage was not measured/tested by av-c.

 

Applications line Script Defender or Script Sentry intercept the .hta file if you download(dangerous practice) it
The GreenBorder test needs to bypass many security layers in my setup. Some of them requires my intervention so I´m protected against this kind of leak

 

Yes, but what if the malware reuse that code and package in non-HTA file?
I would feel more confident if I can block it regardless of what extension the file is, be it HTA or EXE or JPG or MP3.

 

Okay, so this is the only source of the test, right?
http://www.safensoft.com/security.phtml?c=83&id=1100

A clever reader might think that it may be a scam since even the official AV Comparatives doesn't have the copy.