New Detection Test - Dennis Labs

hi,
this test is ethically and deontollogically corrupted.
And this affirmation for various reasons.

The result 100% is an heresy for security software testing in general and av testing in particular:
It has been shown that av detection is an undecidable and NP complete problem (Cohen, Spinellis).
Then in a mathematical way, it is not possible to reach such result, as in real world.
All tests that provoide 100% result means two things:
a/the testers are incompetent and have not the required skill for defeating the product they test, and in this case they should choose another job or hobby.
b/the methodology is flawed and corrupted by design: it is the case for most tests done with av editors partnerships like vb100 and ICSA in particular.
In our case, Denis Lab (funny, hope that it is a P4 class one) has done the test with the promiscuous collaboration of Symantec/NOrton.
A few months ago, Denis lab has published a test of Norton versus Microsoft av, and some of the url have been submitted by MessageLabs, a Symantec company.
These two test try to demonstrated that Norton av is better than Microsoft and other av.
c/100% result is an heresy because many av Eula notify that the product can not provide 100% detection.
d/Money is not compatible with Independence.

I'm glad to see some literature references on this thread but there is no need to read Seneque, Lao Tseu,Voltaire, Montaigne, Shopenauer, Bakounine or Thomas Mann to understand this:
When Money enter in a room, INDEPENDENCY, HONESTY, OBJECTIVITY, NEUTRALITY and TRANSPARENCY disappears by the window.
You can't be paid for a test that will show how ineffective is the product.
This dilemma has been dramatically demonstrated with the implication of business rating agencies in the financial crisis
http://en.wikipedia.org/wiki/Credit_rating_agencies_and_the_subprime_crisis
The only independent tests are done and (sometimes) published by ethichal hackers, schools/univerities, and government agencies.
What is true for Denis Labs is also true for Cascadia Labs: http://cascadialabs.com/

This things said, this test has the right approach. the "in he wild" scenario.
Even if in the cloud based avs require a specific methodology.

Now what is the impact of this test on serious people?
For my concern, NOTHING, as with most other tests.
I think since a few years that Norton is an overestimated av, that the quality of Symantec research is often more interesting than their products (especially when Peter Ferrie was in the "box"), and most off all:
Norton av is the leader in many countries, not because it is more effective than other products, but by the massive and effective marketing of Symantec (and this test is only a part of this marketing)
One of the key of this marketing is the partnership with PC manufacturers and then the number of preinstalled racket Norton software.
Racket because the law is on my side: http://www.channelregister.co.uk/2009/06/11/symantec_mcafee_settlement/
And there is much to say.

Rgds

 

I agree with the many esteemed users of this forum. The test methodology used is probably better and more real world.
But I still discard this test as advertising propaganda, due to these starkling facts:

• 1: Remember the old saying (now made famous by Mr.Madoff), " If its too good to be true, then its probably not ". Well, the test claims Symantec detected 100 % The day any AV detects 100%, I'll eat my shoes.... But till then its a open secret that no product can envisage a 100% detection and if they do, then its very very suspicious.
• 2: Only 40 active malware URLs !! I as an amateur can easily gather more URLs using various malware site trackers. For a professionals to be able to gather only 40 samples is very surprising. There is much more than 40 strains of unique malware/exploit/phish lurking freely over the web. How can you capitulate 40 samples to claim internet protection supremacy ?
• 3: The devil and truth both exist in detail. Detail is conspicuously missing here ( esp. for rival vendors like Panda). When an "independent" lab usually tests different products they take the vendors into confidence first. Here it seems rival vendors had no clue to this test. Plus after a test is done, usually the result is first sent to all vendors and they are given time to prove/argue over the result before they are made public. So when the results are published the score is agreed upon by all. That's certainly the case with hardware/network testing (that I can vouch for). But in this test, clearly Dennis Labs has not talked to Panda, Avast and others. So the tag "independent" used with this commissioned test is a farce.

 

never agreed more

 

The “100% detection" result for Norton Internet Security 2010 (NIS) is being misinterpreted (and misrepresented) in this thread. Symantec (or Dennis Labs) is not claiming that NIS provides “100% protection” against all possible malware instances -- rather, it is stating the empirical fact that the product detected 100% of the cases included in the test.

Concerning the sample size of 40 cases, it’s obviously true that “more is better.” Yet, remember that the same set of cases was used to compare all products, thereby yielding fair and comparable results. Furthermore, recall that the test was designed to mimic a user’s “real world” interactions with the web in contrast to the typical (and somewhat meaningless) tests in which thousands of “zoo samples” are used in a detection exercise (and for which almost all products score 99%).

Additionally, note that even a competitive anti-virus company has acknowledged that this quantity is sufficient for a credible test: “although their sample set may only be of 40 samples, it is still a valid test” (comment by PrevxHelp in post #136). Panda, also participating in this thread, has never questioned the “40 issue." If two competitive anti-virus companies don’t have concerns with this aspect of the test, then why should anyone else?

 

Symantec openly acknowledged the fact that they paid for the expense of conducting the test -- that’s not news. However, this fact does not necessarily prove that the results of the test would have been any different if the funding was provided by a different source (e.g., by Microsoft). To make the argument sound, it is necessary to demonstrate such an outcome; otherwise, it’s nothing more than innuendo and speculation, directed against one the most prestigious, largest and oldest manufacturers of anti-virus protection in the world.

As a competitive anti-virus company rightfully acknowledged, “Testing organizations don't work for free because of the time it takes and the value which the tests provide. To reiterate - testing organizations don't work for free” (PrevxHelp in post #136).

To borrow a phrase from Al Gore, the former Vice President of the United States, this test by Dennis Labs is “an inconvenient truth” for those whose perspectives of Symantec are fixated and inflexible.

 

Again. Norton is a big name and by that I mean well known. Why use a 3rd world testing company when you could of used a big one that is also well known. Answer is simple You want it to shine a good light on your product not the real light. I have no problem with norton but in real world experience as you keep pointing out I have seen Norton fail more then any other product normally to the point its a reformat reinstall situation. Seeing these test just makes me bust up laughing.

Also to barrow a quote from our former President, Bill Clinton.
"I did not have Sexual relations with that Women" Pretty much Ya I believe you about that much as well.

 

Simple. The "well known" ones don't test correctly. If you have read the thread, you would have noticed one of the "big names" IBK from av-comparatives praise the testing methodology in this test.

 

Fajo, are you speaking of the 2010 edition of Norton Internet Security? True, years ago, the product had some challenges, but the current version seems to have overcome those problems. In particular, note that Norton Internet Security 2010 received the highest rating in the most recent malware removal test conducted by AV Comparatives (October, 2009).

* * * * * * * * * * * * * * *​

Here’s another way to think about this test: convergent validity. If the test by Dennis Labs showed Norton Internet Security 2010 to be an excellent performer while a collection of other tests by different organizations showed the opposite, then there would be more merit in questioning the former. Yet, in this situation, the former is in complete alignment with the latter, suggesting that all are converging upon the same conclusion: namely, that Norton Internet Security 2010 is an excellent product.

P.S.: To quote Bill Clinton, "It all depends on what the meaning of the word 'is' is."

 

Interesting choice of words... you calling others fixated when it's you who can't seem to let this go. Why are you so intent on trying to making others see things your way? Isn't it good enough for you to have your own opinion of Symantec, and let it be? I think Symantec is horsecrap, but I have no burning desire to make you feel the same. To each his own, brother.

 

it seems that the only valid tests are the ones that agree with whatever the poster(doesn't matter which one!) seems to think is correct(pre-conceived ideas!),ones that give results they don't agree with are always "flawed" or even heresy??god some people view anti-malware as some kind of religion now!

 

God you guys keep praising this as its something to be proud of frankly all AV's failed that horribly no one should of got above a average ratting. Just because you got a rating does not mean you did a excellent job all AV's in that section should be ashamed of that.

You could always have them run the test with the same methodology. But again I highly doubt those tests would ever see the light of day. Hence Why you went with a hole in the wall company. "Independent" testing has NEVER been in Norton's best interest.

 

Where do I sign to surrender my freedom of expression.....

Regardless, 100% is utopian. Even if in a minuscule test if you get 100% the sample set validity is very suspicious. Emphasis on suspicious, else my consistent stand could be misrepresented.

I repeat again, as an amateur I can obtain over 40 real-world threats actively hosted and running on the net using various malware info sites. So why can't pros like Dennis Labs get many more ? Thats the point of doubt.

 

The "independent" tag here in question is :

Code:

in⋅de⋅pend⋅ent  [in-di-pen-duhnt]  
–adjective
not influenced by the thought or action of others.

If Dennis Labs is truly independent and neutral, they should have at least informed the vendors whose products they were putting to the test. And got their approval for their test. If they had done that Panda,Avast and others would have no objections.

 

wouldn't you prefer that they DON'T inform any vendor if they are independent?then you couldn't suspect that they may have manipulated the updates to get better results?whatever way any product is tested it seems some people will never be happy or agree with the results,I am not just meaning this test every test that has been run and the results discussed on here has had members claiming the test is flawed in some way,probably because their fav product got less than a glowing report,its the way of the world I guess:-read report but only agree with it if it agrees with you!if it doesn't then obviously there is something wrong with the test!

 

Can't the same be said for Norton. I guess in the way that's the point.

 

The problem is that while the educated users of Wilders will realise that to be true the impression given to the average users seeing that page will be exactly that...this product offers 100% detection.They won't take into account the tiny number of samples involved,to them 100% means just that and that is exactly the impression Symantec intended to give.

 

you just don't get what I'm getting at do you?your comment just really proves what i mean
I have a feeling that no mattter how independent a test was , no mattter the technique and no matter the sample size used,if the resultsts didn't tie up with your pre-conceived views on the products tested then that test would also be flawed in your eyes:-think it is called being biased in your views

 

Explain how I "Don't get it" Because your comment states if Vendors knew about the test they would "maybe" try to manipulate the results. Why cant the same be said for Norton they knew the test and what it would consist of, Why could they not be accused of the same ?

 

I wasn't just meaning this test but you took what i wrote and used it to have another go at norton!doesn't that say more about your biased view on that product rather than anything else?

 

Frankly your comment can be interpreted both ways, Just because you choose not to see it that way means nothing of what other people see it as. Also, How could I tell you did not mean this test ? You posted it in this thread and about this subject.

As for Biased view I have nothing against Norton, they act the same way most AV company's act stuck on them self's and think they are the best thing sense sliced bread. What I do have a problem with is how they think this is a "Independent View" and a "Reputable Source" It's amazing to me how when a company pays for a test they always try to find the most unknown hole in the wall company they can find.

In the end this test was paid for for one reason, Marketing no company spends money on something unless they expect to at least make it back. That's just not good Business otherwise.

 

I’m sure it is not difficult to secure more than 40 cases to test -- but, I’m also sure that it increases the cost of the testing, which is probably the real constraint. Any other anti-virus vendor can replicate the methodology used by Dennis Labs, choose the number of the samples they wish the testing company to select, and pay the corresponding fee. So, rather than wondering why Symantec “only” used 40 in this simulated “real world” scenario, perhaps the better question is why the competitors of Symantec haven’t used any to-date?

Alternatively, have those competitors already replicated this test using the same methodology, and are suppressing the publication of the results because their products' performance is unfavorable? Of course, there is no evidence to suggest this is true, but apparently evidence is optional within this discussion.

If the other vendors agreed to pay part of the testing fee, then yes -- they would have a right to voice their concerns both before and after the result. However, I think that makes the test less (not more) “independent.”

I don't see why any anti-virus vendor needs to provide "approval" for an organization to test their product.

Diversity of opinion is a good thing, provided that those opinions can be logically argued and empirically supported; otherwise, as you say, it’s just “bias.” Simply asserting that Symantec “manipulated” the selection of the anti-virus products tested, the selection of samples used in the test, or the actual results of the test seems to have no foundation in fact whatsoever, based upon my reading of the posts in this thread. If I’m mistaken, then I’m more than willing to change my viewpoint.

I think it is quite fair to document and discuss the limitations of the testing methodology, but attacking the character and integrity of the sponsor company without cause seems very inappropriate, indeed. I would make the same argument in support of Panda or Prevx or any other major anti-virus company, if the roles had been reversed and one of those companies had paid for this test, regardless of the results obtained.

 

If you had removed your blinkers for even a few seconds you would have noticed that my post was an observation on on the reaction that forum members seem to have to all tests
As for any AV company believing they are the best thing since sliced bread:-If they don't think their product is the best or one of the best who will?
I very much doubt if any company advertised their product as being 2nd rate they would get many sales,do you?

 

I have always questioned Prevx's marketing strategy as well. Same as Symantec, I don't believe a bit on them

 

I couldn't agree more . I guess Wilders' just doesn't forgive that easily; many of them remember being burned [more than] once before by Symantec, and are quick to pass judgment. The ironic thing is, a few of these same people, I recall, have praised Symantec for Norton 2009 and 2010 .

Someone once told me to "hate the company, not the product" .

 

Pretty arrogant statement, don't you think?