PassMark: Consumer Antivirus Performance Benchmarks 2011

A report assessing the performance of internet security applications, authored by PassMark, has been issued: Consumer Antivirus Performance Benchmarks (2011).

 

Another one sponsored by Norton (4 in one day) I am losing my faith

 

just cause norton scored best doent mean it is sponsered by symantec

nw u guys are even doubting av-test, etc then god help ur trust

 

Yes, sponsored is annoying, as we'd all love to see some independent tests. However, there's no getting away from the fact that NIS2011 is one of the lightest, if not the lightest suite out there - just because a test is sponsored does not mean that its results aren't valid.

More like Symantec realises that their massive overhaul needs to be marketed, and from their point of view, rightfully so. Good choice to have Passmark commisioned - they are generally thorough.

 

Thanks Pleonasm for update. I do not believe in such sponsored tests.

 

I didnt invent anything is written in the report (at the end) that Norton paid to PassMark for this test and Norton gave to PassMark some of the test scripts used for the tests.

 

So what if the test was sponsored, and Symantec gave scripts?

It's results correlate with av-comparatives' performance tests, and a number of other tests. Are you suggesting this test is not trustworthy, in context of other tests that we have seen?

Several independent organisations drawing the same conclusion is pretty powerful evidence, wouldn't you agree?

 

Very good work Eset.

 

Yes, it is called “convergent validity.”

 

Based on the test date of some of the software and the version number of some of them this test is completely worthless, to me anyway.

 

Really? Seems like it is pretty recent builds from what I saw.

 

Most tests funded by a company are terribly biased. For example, last years report (funded by Symantec) also pointed Symantec out as the winner. Or this one: http://www.passmark.com/ftp/Fast and Effective Endpoint Security for Business - Ed1 June 2010.pdf , funded by Eset, guess who won?

 

i would not exclude probability of biasing with company funded but still i see i would agree that eset deserve to win that time

 

This gives them the ability to e.g. initially do the test on a larger set of products, and then (after seeing the results) only have the final report include the products that actually scored worse than Norton.

It's just my guess but looking at the list of tested products, I just find it weird... where are the other products that were included in the previous PassMark tests?

 

One can wonder why they left out testing MSE. Could it perhaps be because it doesn´t impact on the performance at all and IMHO probably would have aced this kind of test? Just a thought...

Regards,
Joakim

 

My point was not that the methodology used was good (which it clearly isn't, an independent test would have been far superior, and this definitely looks like marketing), but that in context of other performance reports, Symantec's performance in this test does not look out of place (despite them submitting scripts). I can't comment on other vendors though, what are your thoughts?

 

I doubt it.

 

This test is sponsored by a given company . However , think about other independant tests sponsored by all vendors who participate.

We as end users can never be sure that a vendor didn't pay to someone or to an organisation to get better results . For example - AV-Comparatives , we know they all pay , but how much do they pay remains unknown to the public . It is also not clean if they all pay the same amount of money or some pays more to have more privilages / higher results ?!

 

Smells like an advertisement for Symantec to me.

 

Correct. However, it would be in the vendor’s own best interest not to do so, because the short-term ‘gain’ would be more than offset by the long-term ‘pain’ resulting from lost business due to a damaged reputation, if the activity were discovered.

 

Oh , that would be behind the scenes - the vendor and the testing org. will only know.

 

Symantec funded this farce, it's plain stupid advertisement.
I bet they also pay for laughter when their CEO tells a joke.

Cheers

 

Just thought I would add my 2 cents, as one of the reports authors.

We do benchmark testing for companies who commission them. It is understandable to think that because company X paid for a report it will be bias in their favour.

However we do performance testing for about a dozen different security companies. Can't name them all as we are under NDAs.

The majority of the reports never get made public. They are presented to the company that commissioned the report for their internal use to improve their marketing and technical development.

So the reports that do get made public are typically only the ones that favour the company who commissioned the report. Unfortunately this can give rise to a perception of bias, as the public doesn't see all the reports that paint the commissioning company in a poor light.

The test scripts referred to above are just DOS scripts to copy files between folders, or unzip/zip a large zip file from the command line.

The reason for using scripts like this is becuase Neil J. Rubenking (the well known reviewer a PCMag.com) also tests using similar test methods and the companies all want to rate well in Mr Rubenking's tests.

Mr Rubenking has had a significant impact on how the industry looks at AV performance. And it is not suprising (to us) that Norton also did OK in the PCmag testing.

The rest of the tools are public tools like our Apptimer the Microsoft xperf tools and the 3rd party Httpwatch.

So pretty much anyone with a few weeks of spare time should be able to reproduce the results, and I would encourage people to do their own testing to confirm the results.

 

Are you even aware of who Passmark is ? If Symantec had rigged the test, then they should have won ALL tests. Notice that that lost quite a few test but over all came out on top.

 

Why? Who pays the piper calls the tune...

Passmark http://www.passmark.com/forum/showpost.php?p=9460&postcount=7

Passmark will show you whatever the customer wants you to see.

Cheers