New Detection Test - Dennis Labs

To clarify, note that Symantec did not pick the test samples for the test -- rather, the independent testing organization selected the samples (see page 11 of the report).

The definition of “compromised” used in the report is “Malware continues to run on an infected system, even after an on-demand scan” (page 16). I believe the intent of the use of the term “compromised” is to denote a case where the malware bypassed the anti-virus product to at least the point where it was installed on the PC.

I’m not sure I understand the issue. The report defines “defended” as “Malware was prevented from running on, or making changes to, the target” (page 16) -- it seems quite clear. If Norton Internet Security 2010 successfully “defended” the PC, then there would be no alert/effect/threat report for a scan.

Note the repetition of “None” for case 5 for AVG, for example, indicating that the same standards were applied across the set of tested products.

Unfortunately, these results are for the 2009 (not 2010) version of Norton Internet Security, and thus are not applicable to the current discussion.

Provided that the PC configuration was constant for testing all anti-virus products (which it was), then the test is fair and the results are not impacted.

Again, note that Symantec did not pick the test samples for the test -- rather, the independent testing organization selected the samples (see page 11 of the report).

According to the report, the test “was configured to allow access to the internet so that products could download updates and communicate with any available ‘in the cloud’ servers” (page 14). This same procedure was implemented for all anti-virus products tested, giving none an advantage, and thus seems completely fair -- correct?

 

Exactly. I am seeing he is asking questions that are just getting ignored. I would like to see them answered also, even though I don't use his product. The lack of a direct response to them is taking even more credibility from the test. Perception is important, and if people perceive that a paid for test will favor the person writing the check, it doesn't matter if it is correct or not. I am not suddenly going to switch to NIS because they score perfectly on every test the have ever paid for.

 

Well, I was refering to this remark of Pbust

 

This once again proves the point that for the immature technologies that mostly depend on the AV signatures, VirusTotal is a good venue to prove their point. Avast doesn't have a clue and to think that blog was posted in their CEO corner, proves that cluelessness goes all the way to the top!

 

These are not real every day user in front of the face threats! They need to get some UAT in there create a Model Office Environment to Simulate End User at Home faced with the disaster threat on desktops and laptops. Test, test and more test what is all this going to show us nothing! Every test is based on which software package is better. Norton always going to be on top Symantec has invested too much into the Security Threat Business since their NAV for DOS then NAV for Windows 3.1. They got all of corp America using Symantec End Point Protection, McAfee, TrenMicro OfficeTrend an etc. Still threats do get in the Admin suppose to alert everyone that one system on the domain got infected then he or she sends out a ticket to get the AV off the System.

 

There is humor in the rant, but unfortunately I don’t see any technical merit in this blog that adds value to the discussion. Maybe someone can extract and summarize any worthwhile content from the blog, and post it within the current thread?

Ah, Pbust raises a valid point that deserves clarification. Nonetheless, I’m not sure what would represent a better testing alternative. A test needs to hold constant as many variables as possible in order to ensure the integrity of the results, and the methodology used “a close simulation of a live internet connection and allows each product to experience the same threat” (page 12 of the report).

Of course, if Panda believes that this is the source of their poor rating in the test, they should correct the deficiency, repeat the process, and publish the results in order to prove their point.

 

Perhaps they need to make a donation first in order to get some sponsored answers

 

Maybe you just misunderstood what it said. The whole point was that if you commission a test, you make the rules and obviously have thus control over the results of the test.

By the way, your statement that "avast doesn't have a clue" is a strong one and would deserve an explanation...

 

Using HTTP based C.AV with Squid (transparent) web proxy in (offline) mode!.. isn't a flaw ?

 

I just agree... No more comments...

 

Scene: Henhouse
Player: Fox
Activity: Guarding

 

Yes, Symantec influenced the design of the test -- that was the objective. From Symantec's viewpoint (which I share), many anti-virus comparatives lack realism. The methodology they created is, I believe, better than most in simulating a real-world scenario. Of course, all testing methodologies -- because they are simulations and constrained experiments -- have limitations.

It does not logically follow that Symantec has "control over the results of the test," however.

For the sake of argument, let’s assume that you are correct: Symantec selected the test samples. In this case, Symantec would knowingly be engaging in deceit. Are you prepared to both make this assertion and support it with something more than humor?

 

Like what?

 

Trash test. I'm hoping that IC takes a shot at them sometime soon.

 

I’m flexible: present any evidence that you believe is trustworthy to support the accusation that Symantec knowingly rigged the results of the test by selecting the samples.

* * * * * * * * * * * * * * *​

It’s disappointing to see the reputation of Symantec attacked in this thread, based solely upon conjecture and innuendo. Obviously, the test sponsored by Symantec has limitations (as do all other anti-virus comparatives), but I believe Symantec is pushing the testing of anti-virus products in the right direction -- namely, toward more realism. If nothing else, they should be congratulated for that accomplishment.

Hopefully, other vendors and testing organizations will improve upon the methodology used in this test, thereby encouraging competition which in turn drives innovation that results in improved solutions for users.

Peace.

 

First of all, I didn't say they selected the samples. You said that. Read what I wrote. I said, in effect, that the fox is guarding the henhouse, which can easily be interpreted to mean that when Symantec pays, the "independent" organization can come up with results favorable to the "employer".

But I'm as flexible as you are, Pleonasm. And I recommend that you not ask others for something that you yourself can not provide.... evidence. I'd like you to "present any evidence" that supports your assertion that what I said is not correct. You can't. You are operating on faith, Pleonasm. Faith that Symantec would not engage in deceit and faith that because an organization describes themselves as "independent", that they really must be.

I'm operating on faith too, Pleonasm. I have faith that such testing must be at arm's length, i.e. conducted at a distance sufficient to exclude intimacy. Paying for a test does not fit that definition.

 

I'll have to rely on my memory: SP2 (non-English version) includes Java. Or is it just the shockwave player ? I'm not sure.

Anyway, my Windows XP CD predates SP2, so I can't be certain.

 

If I remember correctly, Windows XP SP2 removed the Microsoft Java Virtual Machine component--Microsoft's implementation of Java VM.

 

AFAIK SP1a onwards Microsoft no longer included Java VM due to some legal issues (in Windows XP). But this is getting off topic.......

 

Yes, you are effectively implying that Symantec is involved in some kind of deceptive practices. And Pleonasm is correct.. where is your proof

You cannot be going around maligning well respected companies that have been in this business longer than anybody else and assuming they are guilty until proven innocent. Not sure which world you live in where folks are guilty until proven innocent. Last I checked its the other way round.

Also, and I think most importantly, its not the results of this test that count, its a company taking the stand that they will not put up with this absolutely ridiculous flat scanning tests from av-test and av-comparatives and instead attempt to pay for someone to do a real world test. Believe me, NO ONE is willing to pay for these tests which I am sure cost fortune when compared with scanning 5 million samples.

If you want to make a positive contribution dont criticize the results, instead critique the testing methodology so it can be improved and that way everyone can benefit from better tested security software. AMTSO has been trying real hard to get testers to test correctly, and in all fairness to all testers they all want to do the right kind of test BUT ONLY if someone is willing to pay them for it.. and no one is. Certainly not these magazines that are on a shoestring budget. So the security companies are forced to pay for the testing themselves.

And mark my words.. you are going to see a lot more real world tests like this, so get on the bandwagon and suggest improvements and make a long-lasting difference

 

Yes, I agree. However, my “faith” is based on the reputation of the company which, to the best of my knowledge, has historically operated in manner that is very respectable (but, like all companies, not perfect). It isn’t “blind faith.”

Obviously, it is best if testing is done at an “arm’s length." However, the fact that Symantec is paying for the services of the testing organization doesn’t necessarily imply that wrongdoing has occurred, which seems to be the undercurrent of several posts in this thread.

At least in the United States, one is innocent until proven guilty. For this reason, one doesn’t need to “prove innocence,” as you suggest when stating that I need to submit evidence of proper behavior; rather, the burden of proof falls completely upon the party seeking to prove guilt. (Asking for evidence, therefore, isn't a symmetrical situation.)

Your comment about the fox and the henhouse was made as a direct reply to my citation that the samples were not chosen by Symantec (see post #62); hence, the inferred connection. However, if this was not your intention, then I accept you at your word and apologize for my misunderstanding.

 

No, Pleonasm and I are NOT related.. but we do think alike

 

I do not use Norton products so I have no dog in this hunt. My opinion is that if Norton is "suggesting" that the tester use samples that could help the outcome in their favor, they would have alot to lose buy doing that. If one could prove they did that, they would lose all credibility! Who would trust them in the future? It would be a dangerous road to go down. Norton has done well in most testing & I have a hard time believing they could afford to rig any such test.

You may not agree with how the test were done & how the samples were found but that does not mean they had anything to do with it, even if they paid for it.

 

Pleonasm, I sincerely doubt that I can change your mind about anything we have discussed, and I know that you won't change mine. I'd like to continue to keep this discussion civil and have it end on a positive note... so thanks for sharing your thoughts on Symantec.

 

If Symantec sponsored the test then the results are presumptively questionable.

And I think the results of this test are to be marketed toward the consumers currently using free antivirus programs. Surely nobody can justify a quality internet security suite being tested against four stand alone av's. But discounting the effectiveness of free antivirus programs is what Symantec has been squawking about since MSE was released. And bragging about how NISS beat out the free programs is what this test is all about.

It would have been more interesting results, and a more valid test, had a quality free firewall/HIPS program (ie free Online Armor, etc) been ran alongside the free antivirus programs.