Security Products Performance Benchmarking (Sep 2010)

[lordraiden]

Interesting comparative:

Document: http://www.passmark.com/ftp/antivirus_10-performance-testing-ed1.pdf

Overall scores: http://img89.imageshack.us/img89/6439/69297820.jpg

[Ade 1]

How long did those tests take to do just to inform us what we knew already?!

I guess some people have too much time to burn.

[subset]

From the PDF... (Page 20)

-{ Quote: "Disclosure
Symantec Corporation funded the production of this report and supplied some of the test scripts used for the tests (See Appendix 1 – Test Method – How did we conduct these tests? below)." }-

Cheers

[dawgg]

Still looks like a good, indepth test.
Not sure about some of the benchmark tests though, my comments/opinions bellow...

4 - there's such a short time-peroid in it, it doesnt matter, especially considering the number of times the average user opens the UI.
5 - performance can be seen with other tests rather than idle mem usage which is only a number, that doesnt matter, time matters.
8 - you install it a few times a year, its not the end of the world if it takes a couple of min longer.
9 - nothing to do with performance, and doesnt show the peformance impact on the computer, although Trend and G-Data are considerably high relative to the others.
10 - nothing to do with performance and quiet irrelivant - as long as the registry keys are cleaned up after uninstalling and they dont cause any problems, most users dont care.


Edit: and no Avira? - this is going to be horrific for some users here

[salmol]

It seems that they tested F-Secure IS 2009 and not IS 2010.

[raven211]

-{ Quote: "It seems that they tested F-Secure IS 2009 and not IS 2010." }-

Correct - easily seen in the summary.

[flyrfan111]

Wow, the company that funded the test WON!

[dawgg]

-{ Quote: "Wow, the company that funded the test WON!" }-
Would you prefer it if they came second? ;D

[firzen771]

-{ Quote: "Would you prefer it if they came second? ;D" }-

i wuldve prefered if it wasnt funded by them, OR the people who fund it arent allowed to participate (which wuldnt really work out)

[IBK]

the one who funds such a test usually keeps the right to decide what happens with the report, i.e. if it gets published or not. as the outcome of the test was good for the funder, of course they decided to get it published. that's why you will almost always see the one who funded the report coming out first (or very good). its not because the results are faked in favor of the funder, in this case its because the funder decides about what gets published.

[firzen771]

-{ Quote: "the one who funds such a test usually keeps the right to decide what happens with the report, i.e. if it gets published or not. as the outcome of the test was good for the funder, of course they decided to get it published. that's why you will almost always see the one who funded the report coming out first (or very good). its not because the results are faked in favor of the funder, in this case its because the funder decides about what gets published." }-

hmm so does that happen at AV-C as well?

[trjam]

very true IB, you pay for it you better think they have a say. This may very well be a somewhat valid test and everyone is so focused in on who won and paid for it. What I find interesting is number 2, Eset. If you take the tests for valid, they did an outstanding job.

[dawgg]

-{ Quote: "i wuldve prefered if it wasnt funded by them, OR the people who fund it arent allowed to participate (which wuldnt really work out)" }-
Ignore Symantec's results then and only look at the others... (unless you think Symantec may have fixed the tests so Eset comes second and McAfee came last) - go figure, none of us know about this for certain. I personally take this test as a reliable indicator more-or-less.

... and someone's going to have to fund it, especially if its going to be released in the public domain for free - I guess AVs have the most interest in it.

[firzen771]

-{ Quote: "Ignore Symantec's results then and only look at the others... (unless you think Symantec may have fixed the tests so Eset comes second and McAfee came last) - go figure, none of us know about this for certain. I personally take this test as a reliable indicator more-or-less.

... and someone's going to have to fund it, especially if its going to be released in the public domain for free - I guess AVs have the most interest in it." }-

well how does AV-C run? they seem to be pretty reliable and unbiased? im actually kinda curious since im not really sure how AV-C does it.

[dawgg]

-{ Quote: "well how does AV-C run? they seem to be pretty reliable and unbiased? im actually kinda curious since im not really sure how AV-C does it." }-
Think their pricing structure is more reliable - its fee the tested AVs have to pay.

[trjam]

IBK has stated many times in the past that yes, individual testing for a specific product does come down to what the vendor decides in the end. And that is totally the way it should be. When it comes to the overall tests, yes their is a pricing structure but doesnt influence the end results. It cant. Vendors agree to this up front and understand the value of having a testing organization such as his as a useful tool for the future. You do not have to be first. If through his tests you demonstate your product is taking from the results what needs to be done, and you are increasing your products ability, even in small increments, that shows you are serious about creating a solid product.

Hence, those who have dropped out in the past only proved they failed at being able to do so. Regardless of what others may tell you. Look at Norman. I have more respect for them for sticking it out to find ways to openly look for ways to build a better product.

[Macstorm]

-{ Quote: "Wow, the company that funded the test WON!" }-
unbelievable ::) ;D

[renegade08]

Guys, this is performance test.
Who cares about: scan speed of solid state drives, installation time, installation size, file format conversion ( :blink: ).

-{ Quote: "Benchmark 14 – File Format Conversion
This test measures the amount of time required to convert an MP3 file to a WAV and
subsequently, convert the same MP3 file to a WMA format. " }-
If this test was suppose to make laugh, well it did. (No disregards to the testers(who were doing their job) or the test itself).
I don't think that this deserves so much attention IMHO.

[smage]

It seems that Avira is so good that testers have to change the rules or simply remove the product from the test so that other products get a chance to claim first place.