Whither Testing PrevX ??

Page42 · Feb 26, 2010

Pleonasm said:

At the risk of being repetitive, I will say again that Prevx’s claims of being the "world’s best" may in fact be completely accurate. But, it appears that no one knows the extent to which those claims may be true -- or, if they are "just marketing."
Click to expand...

At the risk of being repetitive? Man, that ship sailed a long, long time ago.

PC__Gamer · Feb 26, 2010

Pleonasm said:

What are you referencing on the Virus Bulletin website? Can you please be more specific and provide a link? (Thanks.)

I agree that a healthy dose of skepticism is always wise when reviewing test results. Yet, organizations like AV-Comparatives and AV-Test seem to have respected reputations in the industry for excellence and impartiality.

I encourage Prevx to participate in one of their whole-product dynamic tests and, as I have said many times in other threads, I suspect that Prevx would perform “well” in these comparisons. How well, however, is unknown -- to you, to me, and to everyone who is interested in an objective assessment of the protection provided by Prevx.
Click to expand...

here you go: I was mistaken and it wasnt on the VB website, but the Viruslist website instead.... i still found it though

Have you ever found a false positive when uploading a file to a website like VirusTotal? Sometimes it happens that not just one scanner detects the file, but several. This leads to an absurd situation where every product which doesn't detect this file automatically looks bad to users who don't understand that it's just false positives.

Sadly you will find the same situation in a lot of AV tests, especially in static on-demand-tests where sometimes hundreds of thousands of samples are scanned. Naturally validating such a huge number of samples requires a lot of resources. That's why most testers can only verify a subset of the files they use. What about the rest? The only way for them to classify the rest of their files is using a combination of source reputation and multi-scanning. This means that, like in the VirusTotal example above, every company that doesn't detect samples that are detected by other companies will look bad - even if the samples might be either corrupted or absolutely clean.

Since good test results are a key factor for AV companies, this has led to the rise of multi-scanner based detection. Naturally AV vendors, including us, have been scanning suspicious files with each others’ scanners for years now. Obviously knowing what verdicts are produced by other AV vendors is useful. For instance, if 10 AV vendors detect a suspicious file as being a Trojan downloader, this helps you know where to start. But this is certainly different to what we're seeing now: driven by the need for good test results, the use of multi-scanner based detection has increased a lot over the last few years. Of course no one really likes this situation - in the end our task is to protect our users, not to hack test methodologies.

This is why a German computer magazine conducted an experiment, and the results of this experiment were presented at a security conference last October: they created a clean file, asked us to add a false detection for it and finally uploaded it to VirusTotal. Some months later this file was detected by more than 20 scanners on VirusTotal. After the presentation, representatives from several AV vendors at the event agreed that a solution should be found. However, multi-scanner based detection is just the symptom - the root of the problem is the test methodology itself.

Unfortunately there isn't much AV companies can do about it, because at the end it's magazines that order tests - and if they can chose between a cheap static-on-demand test using an impressive-sounding 1 million samples (some of which are several months old) or an expensive dynamic test with fewer, but validated, zero-day samples, most magazines will choose the first option.

As I've mentioned above, AV companies as well as most testers are aware of this problem, and they aren't too happy about it. Improving test methodologies was also the reasons why two years ago, a number of AV companies (including us), independent researchers and testers founded AMTSO (Anti-Malware Testing Standards Organization). But in the end it's the journalists that play the key role. This is why we decided to illustrate the problem during our recent press tour in Moscow where we welcomed journalists from all around the world. Naturally the goal was not to discredit any AV companies (you could also find examples where we detected a file because of the multi-scanner's influence), but to highlight the negative effect of cheap static on-demand tests.

What we did pretty much replicated what the German computer magazine did last year, only with more samples. We created 20 clean files and added a fake detection for 10 of them. Over the next few days we re-uploaded all twenty files to VirusTotal to see what would happen. After ten days, all of our detected (but not actually malicious) files were detected by up to 14 other AV companies - in some cases the false detection was probably the result of aggressive heuristics, but multi-scanning obviously influenced some of the results. We handed out all the samples used to the journalists so they could test it for themselves. We were aware this might be a risky step: since our presentation also covered the question of intellectual property, there was a risk that journalists might focus on who copies from whom, rather than on the main issue (multi-scanning being the symptom, not the root cause) But at the end of the day, it's the journalists who have it in their power to order better tests, so we had to start somewhere.

So where should we go from here? The good news is that in the last few months, some testers have already started to work on new test methodologies. Instead of static on-demand-scanning they try to test the whole chain of detection components: anti-spam-module -> in the cloud protection -> signature based detection -> emulation -> behavior-based real-time analysis , etc.. But ultimately, it's up to the magazines to order this type of test and to abandon approaches that are simply outdated.

If we get rid of static on-demand-tests with their mass of unvalidated samples, the copying of classifications will at least be significantly reduced, test results will correspond more closely to reality (even if that means saying good bye to 99.x% detection rates) and in the end everyone will benefit: the press, the users and of course us as well.
Click to expand...

Do not be fooled by such tests in the past, they were utter tripe and the results published were as false as one could make them.

there will be an argument that 'its still better to detect', but these tactics used have been purely to detect-well in these tests, they help sales, everyone goes home happy.

but when actual protection is needed, alot of these AV's fail...

People are soooo-easy to manipulate if information is posted in a professional manner, from so-called professional testers on a ('repected'?) website,

alot of fools.

however - dynamic testing has begun, and we shall see how things now go.

PrevxHelp · Feb 26, 2010

Thank you all for the kind words! We are honored to be among the few products who have achieved Platinum status within WestCoastLabs and hope to continue our relationship with them to include new and exciting features into the testing mix

Pleonasm said:

We agree! As I stated previously, I suspect that Prevx really would do well in a comparative analysis -- but none of us knows, at this time, how well it would fare against the competition. Such an independent assessment of the performance of Prevx might allow the company to substantiate the “world’s best” and “unmatched” claims under discussion here.
Click to expand...

Although all of the results are not made public to all users, relevant parties know the strength of the Prevx solution. To quote the WestCoastLabs article:

Prevx 3.0 has passed all the appropriate certification criteria. This includes Anti-Malware Dynamic certification, which is constructed to validate products using real-world methodologies that meet the modern day demands for anti-malware testing. The solution is also enrolled in West Coast Labs' Real-Time Testing program, where it is a consistently high performer.
Click to expand...

A comparison of Prevx versus virtually every other security product lends quite a lot of support to our claims as well: http://www.raymond.cc/blog/archives...age-antivirus-and-internet-security-for-2010/

And for links of other reviews of Prevx, it would be worth taking a look at: https://www.wilderssecurity.com/showthread.php?t=244969

WestCoastLabs is one of the oldest and most trusted antivirus testing organizations, and I believe the only one which is accredited internationally as a scientific testing lab. WestCoastLabs' dynamic/realtime testing is of significant value to us - they have a worldwide network of honeypots and users collecting samples which are thrown against the leading AV engines, including Prevx, on a 24/7 basis. This gives us a very precise indication of the strength of our technology and because of our Platinum status with WCL, we currently don't see the need to add more "badges onto our vest" for AV testing. We've proven our effectiveness and will continue to up the game with antimalware protection. We conceptually do not agree with AV tests that show products achieving 99+% detection rates as that is obviously not representative of real world performance, so until we see other tests which are mutually beneficial and can accurately represent the strengths and weaknesses of the products being tested, I don't see us using a few of the other testing organizations.

Habakuck · Feb 27, 2010

What about the Whole Product Dynamic Test by av-comparatives?

I think it is quite good and i would be pleased to see PrevX there.

pling_man · Feb 27, 2010

When is the certification logo going on the PrevX website.

Pleonasm · Feb 27, 2010

PC__Gamer said:

here you go: I was mistaken and it wasn’t on the VB website, but the Viruslist website instead
Click to expand...

Thank you for providing this information.

Yes, I remember seeing an article about that same issue. Please note, however, that the problem being described applies primarily to static on-demand tests. The new whole-product dynamic testing methods conducted by organizations such as AV-Comparatives and AV-Test are considerably more “real” and seek to more closely mimic the actual experiences of users. As the article says:

So where should we go from here? The good news is that in the last few months, some testers have already started to work on new test methodologies. Instead of static on-demand-scanning they try to test the whole chain of detection components: anti-spam-module -> in the cloud protection -> signature based detection -> emulation -> behavior-based real-time analysis , etc.. But ultimately, it's up to the magazines to order this type of test and to abandon approaches that are simply outdated.

If we get rid of static on-demand-tests with their mass of unvalidated samples, the copying of classifications will at least be significantly reduced, test results will correspond more closely to reality (even if that means saying good bye to 99.x% detection rates) and in the end everyone will benefit: the press, the users and of course us as well.
Click to expand...

This isn’t a recommendation to abandon testing of anti-malware products, nor is it a condemnation of the testing organizations -- rather, it is advocating improved testing methodologies, a position that I personally believe is well founded.

I completely agree with your conclusion: “Do not be fooled by such {static on-demand} tests in the past, they were utter tripe and the results published were as false as one could make them.” The new class of whole-product dynamic testing methods, however, change the game and better answer the question that is really of interest to many users: “How well will a specific anti-malware product actually protect me?”

Pleonasm · Feb 27, 2010

PrevxHelp said:

We conceptually do not agree with AV tests that show products achieving 99+% detection rates as that is obviously not representative of real world performance, so until we see other tests which are mutually beneficial and can accurately represent the strengths and weaknesses of the products being tested, I don't see us using a few of the other testing organizations.
Click to expand...

Yes, it does seems that a consensus is developing: the old, static on-demand testing methods fail to accurately represent the protection provided by an anti-malware product. However, the impressive whole-product dynamic testing methods recently employed by AV-Comparatives and AV-Test do seem to be highly “representative of real world performance.” AV-Test, for example, exposed an Internet connected PC (protected by one of the products being evaluated) to 10 malware threats per day for 60 days and measured the threats blocked. Of course, every test has limitations, but this seems to be an excellent method for assessing “real world performance.”

Question: Why doesn’t Prevx consider this approach to be “representative of real world performance”?

Thank you.

P.S.: I’ll take a closer look at all the links you provided.

PC__Gamer · Feb 27, 2010

Pleonasm said:

Thank you for providing this information.

Yes, I remember seeing an article about that same issue. Please note, however, that the problem being described applies primarily to static on-demand tests. The new whole-product dynamic testing methods conducted by organizations such as AV-Comparatives and AV-Test are considerably more “real” and seek to more closely mimic the actual experiences of users. As the article says:

This isn’t a recommendation to abandon testing of anti-malware products, nor is it a condemnation of the testing organizations -- rather, it is advocating improved testing methodologies, a position that I personally believe is well founded.

I completely agree with your conclusion: “Do not be fooled by such {static on-demand} tests in the past, they were utter tripe and the results published were as false as one could make them.” The new class of whole-product dynamic testing methods, however, change the game and better answer the question that is really of interest to many users: “How well will a specific anti-malware product actually protect me?”
Click to expand...

yes, your right it IS about static detections,

however, these tests went on for YEARS, people believed the crap and believed their product was the best etc.

so, why would someone now trust their new methods?

its fine to say, its now Dynamic Testing, but I aint convinced in Av-Comparatives ability to test against Malware, regardless if to wilders - they are gods in the industry.

in the end, the public trust such websites and its all been falsely played, i feel sorry for people who have relied on them to purchase a product, or to at least 'push-them-in-the-right-direction', it shows for YEARS, they have printed such rubbish, such lies, and products have claimed awards falsely,

... im sure its all a big joke, as long as the testers get recognition and as long as players of this game get their awards and recommendations, everyone goes home happy (except the paying customer)

I do like to test my securitys against new samples, but my PC is extremely hard to infect with Prevx on High/Med/Med,

sure, some new files get through, but as soon as they start doing something, prevx stops it.

and a simple 'scan my computer' quick scan, shows the infections.

c:\program files\adware alert\adware alert.exe [PX5: 25DC899800D2348450D4910AA96D05005871D957] Malware Group: Medium Risk Malware
[BPN] c:\users\168957\appdata\local\microsoft\windows\temporary internet files\content.ie5\tb6r4zdq\player_update[1].exe [PX5: 7E24292600FEC98F3C0F01A1F5D21C00BCD2B622] Malware Group: High Risk Fraudulent Security Program
[Z] c:\users\168957\appdata\local\temp\~~156c9e9.~~~ [PX5: 38A5A2C8047C1547908109CF4E67310053202EC1] Malware Group: Malware Component
c:\users\168957\appdata\local\microsoft\windows\temporary internet files\content.ie5\y4hit3pz\file[1].exe [PX5: 90637F560059303D12AD023183079C00EC0E5276] Malware Group: Medium Risk Malware
c:\users\168957\appdata\local\temp\trdlfsdb.exe [PX5: 3F7762280076F28A54760016BD2C1B00B0559BFC] Malware Group: Medium Risk Malware
[BP] (ACTIVE) c:\users\168957\appdata\local\temp\unataqpv.exe [PX5: BE72C65B00541938320B01F73F509A004F665B9C] Malware Group: Medium Risk Malware
[BP] c:\cleansweep.exe\cleansweep.exe [PX5: BE72C65B00541938320B01F73F509A004F665B9C] Malware Group: Medium Risk Malware
(ACTIVE) c:\users\168957\appdata\local\temp\ihxql.exe [PX5: 3B8C15F700E03C3C6C5601547D916B008E47506F] Malware Group: Medium Risk Malware Dropper
[BP] (ACTIVE) c:\users\168957\appdata\local\temp\vcavbcq.exe [PX5: 6435730B002DC24E04C0021BF03663005038BD63] Malware Group: Medium Risk Malware
[BP] (ACTIVE) c:\users\168957\appdata\local\temp\icpaybw.exe [PX5: 09C8E85F00CBA6DC664501C33FED870052147997] Malware Group: Medium Risk Malware
(ACTIVE) c:\users\168957\appdata\local\temp\kacnwtbl.exe [PX5: BCA7CF8E207DECF54EDD00136D02EA00AA38D16A] Malware Group: Medium Risk Malware Downloader
c:\users\168957\appdata\local\temp\r89tm6iwi.exe [PX5: BCA7CF8E217DECF54EDD00136D02EA00B3918D75] Malware Group: Medium Risk Malware Downloader
(ACTIVE) c:\users\168957\appdata\local\temp\459696687.exe [PX5: A9CA301D00612A929A5F0020F822B700DFA3AE64] Malware Group: High Risk Cloaked Malware
(ACTIVE) c:\users\168957\appdata\local\temp\smss.exe [PX5: A9CA301D04612A929A5F0020F822B7000D8AE993] Malware Group: High Risk Cloaked Malware
c:\users\168957\appdata\local\temp\services.exe [PX5: A9CA301D04612A929A5F0020F822B7000D8AE993] Malware Group: High Risk Cloaked Malware
c:\users\168957\appdata\local\temp\nvsvc32.exe [PX5: A9CA301D04612A929A5F0020F822B7000D8AE993] Malware Group: High Risk Cloaked Malware
c:\users\168957\appdata\roaming\sdra64.exe [PX5: 90637F560059303D12AD023183079C00EC0E5276] Malware Group: Medium Risk Malware
c:\users\168957\appdata\local\temp\2858508000.exe [PX5: A9CA301D00612A929A5F0020F822B700DFA3AE64] Malware Group: High Risk Cloaked Malware
(ACTIVE) c:\users\168957\appdata\local\temp\lsass.exe [PX5: A9CA301D04612A929A5F0020F822B7000D8AE993] Malware Group: High Risk Cloaked Malware
(ACTIVE) c:\users\168957\appdata\local\temp\system.exe [PX5: A9CA301D04612A929A5F0020F822B7000D8AE993] Malware Group: High Risk Cloaked Malware
c:\users\168957\appdata\local\temp\notepad.exe [PX5: A9CA301D04612A929A5F0020F822B7000D8AE993] Malware Group: High Risk Cloaked Malware
[BP] c:\users\168957\appdata\local\microsoft\windows\temporary internet files\low\content.ie5\g5f3a5y6\4yez7irubz[1].exe [PX5: 260939A300FA7029A67500D904217400A2490035] Malware Group: Medium Risk Malware
[BPN] c:\users\168957\appdata\local\microsoft\windows\temporary internet files\low\content.ie5\tgwkthel\player_update[1].exe [PX5: 7E24292600FEC98F3C0F01A1F5D21C00BCD2B622] Malware Group: High Risk Fraudulent Security Program
c:\users\168957\appdata\local\temp\0cf187f3.tmp [PX5: 3F7762280076F28A54760016BD2C1B00B0559BFC] Malware Group: Medium Risk Malware
[BP] c:\programdata\microsoft\windows defender\localcopy\{b5c1118a-1d45-4053-bafd-364cc6183cd0}-4yez7irubz[1].exe [PX5: 260939A300FA7029A67500D904217400A2490035] Malware Group: Medium Risk Malware
Click to expand...

edit:

5 minutes later, scan shows some new detections:

HN] c:\users\168957\appdata\local\microsoft\windows\temporary internet files\content.ie5\tb6r4zdq\setupxv[1].exe [PX5: A48CB27BA07E49DEA2DB64A3FD414300E97EC485] Malware Group: Community.Heuristic
[HN] c:\users\168957\appdata\local\microsoft\windows\temporary internet files\low\content.ie5\87055vni\setupxv[1].exe [PX5: A48CB27BA07E49DEA2DB64A3FD414300E97EC485] Malware Group: Community.Heuristic
Click to expand...

this is why Prevx is sooo hard to test against, its ever-working, ever-mutating to find those infections.

id like to say, that once again, i see nothing on my machine that would lead me to believe i am infected.

PC__Gamer · Feb 27, 2010

on another note, after my latest testing, i did get the following:

now, these were zero day samples and prevx seems to have kept me infection-free which is beyond all expectations.

however, this screenshot ive produced does puzzle me, its a full system scan without smart scanning and 0 new threats detected. (and yes, ive tried Max/Max/Max on my settings too)

however, Hitman Pro says 2 detections and the most puzzling part is Prevx is one of the engines to detect it in HMP.

not too bothered, just curious why this would happen?

no extra running processes, no pop ups, nothing i can see, infact PC just seems as normal, just as quick.

Super!

DavidCo · Feb 28, 2010

Just subscribing to get the answer to the last post by PC Gamer

Pleonasm · Feb 28, 2010

PC__Gamer said:

however, these tests went on for YEARS, people believed the crap and believed their product was the best etc.
Click to expand...

It’s also worthwhile to remember that years ago anti-malware applications were not cloud-enabled, and thus these static on-demand tests (while not perfect) were a reasonable assessment of the protection provided at that time. However, in recent years, almost all anti-malware applications interact with the cloud and, for this reason, the old static on-demands tests are no longer relevant (in my opinion).

The use of static on-demand tests by organizations such as AV-Comparatives was not a ‘conspiracy’ nor was it ‘incompetence’ -- years ago, it was a reasonable testing methodology.

PC__Gamer said:

so, why would someone now trust their new methods?
Click to expand...

Seriously, why wouldn’t you trust the new whole-product dynamic testing methods? I don’t see a critical flaw in this approach -- although, obviously, limitations exist (e.g., number of samples tested).

PC__Gamer said:

I’m sure it’s all a big joke, as long as the testers get recognition and as long as players of this game get their awards and recommendations, everyone goes home happy
Click to expand...

If I understand you correctly, you seem to be suggesting that it is impossible to test today’s anti-malware products in an intellectually sound manner. I just can’t accept the conclusion that knowledge about the relative strengths/weaknesses of competitive anti-malware products is beyond human understanding.

PC__Gamer said:

this is why Prevx is sooo hard to test against, its ever-working, ever-mutating to find those infections.
Click to expand...

I honestly don’t see why other cloud-enabled anti-malware products are any different, and therefore why Prevx can’t be compared to them.

PrevxHelp, perhaps you too can comment on this issue?

PC__Gamer · Feb 28, 2010

Pleonasm said:

It’s also worthwhile to remember that years ago anti-malware applications were not cloud-enabled, and thus these static on-demand tests (while not perfect) were a reasonable assessment of the protection provided at that time. However, in recent years, almost all anti-malware applications interact with the cloud and, for this reason, the old static on-demands tests are no longer relevant (in my opinion).

The use of static on-demand tests by organizations such as AV-Comparatives was not a ‘conspiracy’ nor was it ‘incompetence’ -- years ago, it was a reasonable testing methodology.

Seriously, why wouldn’t you trust the new whole-product dynamic testing methods? I don’t see a critical flaw in this approach -- although, obviously, limitations exist (e.g., number of samples tested).

If I understand you correctly, you seem to be suggesting that it is impossible to test today’s anti-malware products in an intellectually sound manner. I just can’t accept the conclusion that knowledge about the relative strengths/weaknesses of competitive anti-malware products is beyond human understanding.

I honestly don’t see why other cloud-enabled anti-malware products are any different, and therefore why Prevx can’t be compared to them.

PrevxHelp, perhaps you too can comment on this issue?
Click to expand...

its nothing to do with the fact that some now use 'cloud' technology, for years there wasnt a single product tested that used cloud, some have only added this in the past year or so, prevx also has other techniques to detect undetected samples, and im guessing prevx dont trust them to use all-what-prevx-is to get its result.

Why wont Prevx enter the tests, im sure i aint far off the truth by saying they simply dont trust them as testers.. their methods, the way they aquire their samples, there are lots of reasons. (of course, i dont doubt they would admit this, as they are professionals and a buisiness and those types of comments are frowned upon towards a tester im sure)

For years companys have played the game of easy-passes in the static tests, regardless if it shows a customer their true value to protect them, and im doubtfull of their capabilitys to test them dynamically.

ive read through the posts on this forum for past tests and for companys removing themselfs from those tests, and people on this forum slated them for it, yet new news and facts only shows those companys in a higher light in my opinion, and shows alot of the others has complete fraudsters! playing percentage games to gain respect & money is disgusting for the paying customers who 'may have' relied on such tests for guidance.

the methods companys have used for these easy-passes and false figures, only got worse as every test went by, sorta like an avalanche.

but like those, there is an end and that would be for static detections, out with the old, in with the new!

however, will similar practises be performed in these tests, only time will tell.

but i cant honestly say i trust they wont, im hoping they wont, but i have my doubts.

thats just what i think.

anyway - regarding my monthly (or so) test on prevx for zero-day samples, i can honestly say, my computer with Prevx is very hard to infect!

PrevxHelp · Feb 28, 2010

PC__Gamer said:

on another note, after my latest testing, i did get the following:

View attachment 215763

now, these were zero day samples and prevx seems to have kept me infection-free which is beyond all expectations.

however, this screenshot ive produced does puzzle me, its a full system scan without smart scanning and 0 new threats detected. (and yes, ive tried Max/Max/Max on my settings too)

however, Hitman Pro says 2 detections and the most puzzling part is Prevx is one of the engines to detect it in HMP.

not too bothered, just curious why this would happen?

no extra running processes, no pop ups, nothing i can see, infact PC just seems as normal, just as quick.

Super!
Click to expand...

This could likely be due to scan result caching - could you send us a full scan log to report@prevxresearch.com so that I can see if the files were cached before they were marked bad?

Thanks!

Pleonasm · Mar 1, 2010

PC__Gamer said:

Why won’t Prevx enter the tests, I’m sure I aint far off the truth by saying they simply don’t trust them as testers
Click to expand...

You could be right. PevxHelp, your comments?

On a general note, I don’t think it is proper to speak of all anti-malware testing organizations in the same way -- there is a distribution of professionalism in that industry, just as there is in any other industry. Sure, there are some testing organizations that are ‘questionable’ -- but, that doesn’t mean that there are not some which are excellent.

PC__Gamer said:

regarding my monthly (or so) test on Prevx for zero-day samples, I can honestly say, my computer with Prevx is very hard to infect!
Click to expand...

Do you realize that you are performing a ‘lite’ version of the same whole-product dynamic testing methodology executed by AV-Comparatives and AV-Test -- thereby providing support for the hypothesis that Prevx can be legitimately compared to its competition using this approach? You’re exposing your PC, protected by Prevx, to a variety of malware and measuring what is blocked. That’s precisely what AV-Comparatives and AV-Test do when running a whole-product dynamic test.

PC__Gamer · Mar 1, 2010

Lol then why do they need to enter the test, they get it from me for free.

Joe probably trusts me more to test prevx anyway, lmao

PrevxHelp · Mar 3, 2010

PC__Gamer said:

Lol then why do they need to enter the test, they get it from me for free.

Joe probably trusts me more to test prevx anyway, lmao
Click to expand...

To be quite honest - this is very true. We know how well our product performs because we test it on a daily basis against other AVs. If any user really needs to validate it on top of what PC Magazine, Immunity, Gartner, WestCoastLabs, InfoWorld, About.com, Softpedia, numerous independent reviewers online, and our own users say about the product, I imagine they could run the most representative test on their own.

Testing by a third party is useful, but how does anyone know if the testing is relevant to their computer usage? There are scores of ways to get infected and many of them are irrelevant to a majority of the users so testing against these methods or threats does not accurately reflect the power of the AV for that user.

That being said:

On a general note, I don’t think it is proper to speak of all anti-malware testing organizations in the same way -- there is a distribution of professionalism in that industry, just as there is in any other industry. Sure, there are some testing organizations that are ‘questionable’ -- but, that doesn’t mean that there are not some which are excellent.
Click to expand...

... we agree, and at RSA this year, we met up with a number of different AV testers who are now starting to acknowledge the ability to test Prevx because of the changes in testing and should hopefully be able to start testing it against other AVs in the future, although I believe the industry is still waiting on some "official" response from AMTSO to finalize the procedures.

Pleonasm · Mar 3, 2010

PrevxHelp said:

We know how well our product performs because we test it on a daily basis against other AVs.
Click to expand...

Please tell me that you are not referring to the “Threats missed by other security vendors” analysis! More seriously...

Based on those tests, how well does Prevx actually perform?

How are the tests conducted?

What is the rationale for not making the results public?

PevxHelp said:

If any user really needs to validate it..., I imagine they could run the most representative test on their own.
Click to expand...

I agree that personal experience plays a key role in an individual’s assessment of an anti-malware product. However, such an exercise isn’t addressing the theme of this thread: how well does the protection provided by Prevx compare to that provided by its competitors? I think it is safe to say that most users lack the resources to conduct comparative testing well.

PrevxHelp said:

at RSA this year, we met up with a number of different AV testers who are now starting to acknowledge the ability to test Prevx because of the changes in testing and should hopefully be able to start testing it against other AVs in the future, although I believe the industry is still waiting on some "official" response from AMTSO to finalize the procedures.
Click to expand...

Are you referring to the AMTSO Best Practices for Testing In-the-Cloud Security Products? Wasn’t this specification approved by the organization almost a year ago?

Prevx isn’t currently a member of the Anti-Malware Testing Standards Organization (AMTSO). Is this about to change?

P.S.: As always, I find our conversations intellectually stimulating and enjoyable!

P.S.: And, on a jovial note, I’m reminded of the following quote:

“For those who believe, no proof is necessary.
For those who don't believe, no proof is possible.”

JCRUYFF · Mar 3, 2010

can BluePoint Security 2010 Personal Editon and Prevx run at the same time?
why prevx don´t put virus name in my opinion is better.

__________________
Prevx 3.0.5.85 with SafeOnline
Hitman Pro 3.5.4 Build 91 x32-bit
BuePoint Security 2010 v1.0.8.99

PC__Gamer · Mar 3, 2010

Cheers Joe,

It is annoying to me when I see prevx tested in a way that doesn't test it properly, its not a matter of scanning a large collection of samples ondemand, especially untested ones.

Pleonasm · Mar 3, 2010

PC__Gamer said:

It is annoying to me when I see Prevx tested in a way that doesn't test it properly, it’s not a matter of scanning a large collection of samples on-demand, especially untested ones.
Click to expand...

Yes, for any of today's major anti-malware products, this mode of testing is not only “annoying” -- it is also misleading, in my opinion.

Triple Helix · Mar 7, 2010

Very cool! http://www.prevx.com/certification.asp

Congrats again on a fine job! Keep it up!!!!

TH

PrevxHelp · Mar 7, 2010

Pleonasm said:

Please tell me that you are not referring to the “Threats missed by other security vendors” analysis! More seriously...
Click to expand...

No, it doesn't

Based on those tests, how well does Prevx actually perform?
Click to expand...

Extremely well - we still add some fringe detections as necessary, but we are overall consistently pleased with our performance and we are obviously objective in the interest of actually improving our products.

How are the tests conducted?
Click to expand...

Our researchers hunt for malware and websites that contain exploits using PCs with outdated security patches, outdated Flash/Java/Adobe/etc. and try and get infected. Once heavily infected, they install Prevx and see how well it performs. Our current record is 20+ discrete rootkits on top of > 400 individual infections simultaneously infecting a PC - our researchers are relentless Periodically, images are taken from these PCs and tested with other AVs as well to give us a good feel of our competition in a live environment and some of the results are honestly shocking.

What is the rationale for not making the results public?
Click to expand...

How many tests done by vendors about the performance of their product are actually accepted as true when released publicly?

Are you referring to the AMTSO Best Practices for Testing In-the-Cloud Security Products? Wasn’t this specification approved by the organization almost a year ago?
Click to expand...

Yes, but I'm unaware of any organization currently performing all of these tests and I'm not familiar enough with them to say if we've actually gone the route of discussing these tests with the testers yet.

Prevx isn’t currently a member of the Anti-Malware Testing Standards Organization (AMTSO). Is this about to change?
Click to expand...

Probably not - we decided not to join intentionally for reasons that probably shouldn't be shared here.

kasperking · Mar 7, 2010

JCRUYFF said:

why prevx don´t put virus name in my opinion is better.
Click to expand...

i second that... cloaked....medium....high risk seems like the pc has caught an exotic infection

Pleonasm · Mar 8, 2010

PevxHelp said:

Q: Based on those tests, how well does Prevx actually perform?
A: Extremely well...
Click to expand...

I suspected the same, so that’s good to hear. It’s too bad that Prevx isn’t receiving the positive press that would occur if such comparative tests were public. That may be a huge (and missed) business development opportunity for the company.

PrevxHelp said:

Our researchers hunt for malware and websites that contain exploits ... and try and get infected. Once heavily infected, they install Prevx and see how well it performs.... Periodically, images are taken from these PCs and tested with other AVs as well to give us a good feel of our competition in a live environment and some of the results are honestly shocking.
Click to expand...

That is certainly a step in the right direction, but are you not testing detection/cleaning rather than prevention with the competitive products? It seems to me that increasingly the emphasis is shifting toward the latter, relying upon procedures such as the whole-product dynamic testing methods employed by AV-Comparatives and AV-Test.

...And, it's still not clear to me why Prevx believes this whole-product dynamic testing methodology is unrepresentative of "real-world" anti-malware protection performance (see post #57).

PrevxHelp said:

How many tests done by vendors about the performance of their product are actually accepted as true when released publicly?
Click to expand...

Yes, I agree -- that is a serious issue, but consider....

Prevx (like other anti-malware vendors) does not seem to be averse to paying a third-party to conduct tests on its behalf, such as the one done by Immunity (if I understand the Prevx-Immunity relationship correctly). Following that model, Prevx could outline the parameters of a proper test and have a reputable third-party organization independently perform a robust and public comparative assessment.

P.S.: Some individuals may object to a test where a vendor is paying a fee to the testing organization. Personally, I think such a criticism is unwarranted when dealing with highly regarded companies, such as the recent West Coast Labs certification (paid for by Prevx, a fact about which I have seen no objections discussed in this forum).

PC__Gamer · Mar 8, 2010

Pleonasm, vendors pay towards avc aswell, so what's the difference?

Log in or Sign up

Whither Testing PrevX ??

Page42 Registered Member

PC__Gamer Registered Member

PrevxHelp Former Prevx Moderator

Habakuck Registered Member

pling_man Registered Member

Pleonasm Registered Member

Pleonasm Registered Member

PC__Gamer Registered Member

PC__Gamer Registered Member

DavidCo Registered Member

Pleonasm Registered Member

PC__Gamer Registered Member

PrevxHelp Former Prevx Moderator

Pleonasm Registered Member

PC__Gamer Registered Member

PrevxHelp Former Prevx Moderator

Pleonasm Registered Member

JCRUYFF Registered Member

PC__Gamer Registered Member

Pleonasm Registered Member

Triple Helix Specialist

Attached Files:

Capture07-03-2010-12.43.48 AM.jpg

PrevxHelp Former Prevx Moderator

kasperking Registered Member

Pleonasm Registered Member

PC__Gamer Registered Member

Log in or Sign up

Whither Testing PrevX ??

Page42 Registered Member

PC__Gamer Registered Member

PrevxHelp Former Prevx Moderator

Habakuck Registered Member

pling_man Registered Member

Pleonasm Registered Member

Pleonasm Registered Member

PC__Gamer Registered Member

PC__Gamer Registered Member

DavidCo Registered Member

Pleonasm Registered Member

PC__Gamer Registered Member

PrevxHelp Former Prevx Moderator

Pleonasm Registered Member

PC__Gamer Registered Member

PrevxHelp Former Prevx Moderator

Pleonasm Registered Member

JCRUYFF Registered Member

PC__Gamer Registered Member

Pleonasm Registered Member

Triple Helix Specialist

Attached Files:

Capture07-03-2010-12.43.48 AM.jpg

PrevxHelp Former Prevx Moderator

kasperking Registered Member

Pleonasm Registered Member

PC__Gamer Registered Member

Useful Searches