Dennis Technology Labs ~ January - March 2013 tests.

If I may ask, are these customers corporate customers or private clients ?

For as far as I know, it would be somewhat unusual for private clients not to have ANY Windows patches beyond XP3.

For WIN XP (home) I still have IE 7, not that I use it much. I never liked IE 8, and I never bothered with IE9 if it's even available !

Java is a different matter. I have seen plenty of people with outdated versions. In corporate situations that might actually make sense.

There can be very different requirements for private clients vs. corporate customers, especially where large corporations are concerned.

 

Was it the URL of a report?

 

I remember the almost same response when a previous test was posted in Wilders thread.

The statement actually says a lot. It indicates to me that the tests being performed by Dennis Labs are on par with the tests Rubenking performs for PC Magazine. At least Rubenking doesn't proport that his malware testing is anywhere as extensive as those done by the AV test labs. In fact he always makes it a point to include a table showing most AV test lab results for the products he reviews.

Bottom line - the Dennis Labs tests are just one more of the multitude of independant tests appearing these days on the web.

 

Rubenking is merely one reviewer and i certainly would not base a potential security product purchase on his review alone.It would need to be on a more broader spectrum.

Being an average home user myself these tests are totally irrelevant to me.
Honestly how much much of this "tested" malware will i come across.?

I favour usability and resource usgae above detection rates.
If an av causes high cpu usage irrespective of its detection capabilities,it is ditched.

I think we are all just going around in ever decreasing circles when we look at these tests.

 

A mix of both. Usually the IE7 installations are on the larger corporate customers. They are slow to update anything. The outdated Java I have seen on both types of customers. Not sure if it is needed for a particular version of some other outdated software or if they are just slow to get it updated.

 

Somehow contradictory statements over here...

If sponser's or money influence is been talked over here,then even AV-C and AV-Tests are somehow funded by AV companies and then why people trust them and dont trust Dennis Labs and of course then AV-C and AV-Tests can also be biased with the company that provides larger amount of funds,again why finger doesnt get pointed on well known test groups and why only on small test groups

C'mon guys seriously..

When in reality every testing organization works the same way and every test is questionable

 

Like lots on here you probably have preconceived ideas of what the result should be,and any results that don't agree with those ideas are the result of a flawed test,not just by this lab but by all of them:-similar comments are posted regularly after results are made known,perhaps a little digging into the history of some of the major test labs would be in order:-Andreas Clementi was considered a bit of a joke when he 1st started testing and releasing his results due to testing methods etc,none of the major vendors considered him a legitimate source of data,now more folk seem to consider him and his colleagues to be the source of data to rely on,I don't fall into that group!
What I do find amusing is the reaction on sites like this to results,with the fanboys of those that good results saying what a great test/test procedure other fan boys of products that do badly(relatively)deriding the test procedure(its got to be flawed!:-right??)and the results,both parties cannot be correct

 

Sports fans of opposing teams in a game both think the refs are calling against their team. Supporters of opposing political candidates think the media is biased against their candidate.

 

lol
I am not a fan of any AV. when it is good, I use it, when it goes bad, I ditch it, simple as that. Norton was my favorite for about 2 years shortly after the overhaul of their software in 2010, then I noticed something is not right with it. I then switched to Kaspersky. I ditched KAV in 2006 because of its tagging NTFS objectID behavior and had never used it for many years, but now I use it again since they come back to the right direction. So, just like relationships between two countries, there is no forever friends, nor forever foes, there is only national interest. For me it's no forever good av, nor forever bad av, there is only currently good av that I will use.
Simple as that.
By judging ppl without any ground, you are already biased. Not all ppl has something bias in their mind.

 

It really feels like this to me. I'm sick to death of jumping from one AV to the next. Panda Cloud and Bitdefender Free are anything but light on my rig for being touted as light and "Cloud" AV's. So having 200 something days left on my WSA subscription, I installed it again and left the settings on default and boy am I happy I did. This thing is crazy light, I love it. No more herky jerky streaming video or slow loading sites, and apps open and work super fast. This is what an AV should feel like. I'm coupling this with a safe and hardened browser (Chrome) and I'm not looking back until something slips through and I get infected.

 

I have avast 8 free installed.And i certainly see no reason to use anything else.
Sure avast does good and bad in different tests,but this wouldnt put me off the product.

Constant high cpu usage would.

 

AV-Comparatives and AV-Test have both tested with outdated, vulnerable software including Adobe Flash and Java.

March 2013 is the first time AV-Comparatives have run their Real-World Protection Test with updated third-party software. In March 2013 only 422 cases were counted out of 780 for the Real-World Protection Test.

It took years for AV-Test to post a methodology on their website and I think the current one is not adequate enough. (Do their main tests even have third party software installed?). Their methodology mentioned in Symantec commissioned tests are better and is where they mention the installed software.

 

not judging people but stating facts:-you can do research and find out yourself,it isn't just restricted to test results on AVs or anything it is just human nature:-we all have preconceived ideas on most things,if something comes along to question those ideas(call them beliefs if you want) we oppose it:-religion is the best example of this!As for kasperky and the way they tagged files it wasn't a problem while using it but was when you uninstalled it,left loads of crap behind which you had to use one of their "tools" or a 3rd party tool to remove,it wasn't that that stopped me recommending or installing it for customers it was all the bugs in later releases:-you want to install stuff that works 99.9% of the time perfectly you do get tired of calls saying"My pc isn't working right again" to finds out its a prob with the "new av" you have recommended/sold!

 

Yes, both have their issues. AV-Test appears to be the lesser of the two evils. At least they listen to the industry when it comes to static on-demand tests etc. AV-Comparatives thinks they are the experts, which they aren't.

Overall, I think these AV Testing orgs have the entire security industry by the b***s. They do whatever they please and there is no one to regulate them.

Worse of all, its very clear that many vendors are optimizing their products for the test instead of focusing on real user problems.

 

Bingo.

 

First of all, I don't want to start an argument.

Both av-test and av-comparatives are major and widely respected and independent testing organizations.

Their tests are not the tests that any average Joe or magazine can do.

It's easy to find fault with their methodologies. But for as far as I know, they are consistent, objective, professional and for their tests (on-demand/file etc.) they use very large samples.

Perfect ? No. But I'd trust them any day over some Youtube guy or magazine.

Vendors optimizing their products for these tests ? Perhaps, but the better products don't focus on that or they would run into problems in real life.

Fortunately these organizations are not regulated ! Government or bureaucratic oversight works so well ...

Use whichever test you like.

All AVs have major flaws. They use the opposite of default-deny, which can only cause many vulnerabilities and problems.

 

Very good reasoning and very convincing. I like discussions based on fact and logic.

 

[Edited once to add the final point about AMTSO, then edited a second time to make this note!]

Hello everyone.

As usual I have enjoyed the discussion and would like to provide some feedback.

I appreciate that some people reading this thread have preconceived ideas, or even conspiracy theories, that will never be changed by anything that I write here. However, for the record, I would like to answer some of the questions, concerns and both direct and implied accusations that appear above.

Prior associations with Symantec, “money talks”. Dennis is a Symantec affiliate. Mock surprise that Norton comes top.

We have associations with most vendors through our involvement in AMTSO. We have provided services to companies including, but not limited to, McAfee, Trend Micro, Kaspersky Lab, ESET, AVG and Symantec.

We pride ourselves on our transparency and the ethical way in which we conduct our business. We have no secret affiliations with any customer.

One of the reasons the vendors like working with us is because our testing can provide a number of benefits. For example, if a product performs very well then the vendor’s marketing team are happy. However, if we discover a problem then the engineers are able to use the very detailed information that we log to fix that problem.

It would be of very limited use to everyone if we fixed the results to favour one vendor. Such cheating would be fairly obvious to the other vendors, and the favoured vendors would be unaware of problems with their products.

Assumption that most tests used internet malware

Absolutely, and this fact was made clear throughout the report, starting with the introduction. We expose the products to live (internet) web-based threats. Products that block malicious URLs thoroughly will do well, whereas those that rely solely on signatures and heuristics face a greater chance of being compromised.

DTL tests are not on a par with the major test labs

All test labs have different methodologies. It would be fairly pointless if we all worked in exactly the same way as we would be duplicating each other’s work.

Some testers concentrate on running vast number of samples through AV engines. We focus on investigating cases in forensic detail. This means that our sample numbers are much smaller than those found in on-demand tests. The workload is heavy and manual. It takes us six weeks to run a 100-sample test with 20 or less products.

Having said that, when other testers perform ‘real-world’ testing such as we specialise in, they also use small sample sizes that are comparable with ours.

The note that the test is unsponsored was an attempt to knock Wilder’s argument up front. Also, unsponsored is a matter of definition.

Important though your opinions are to us, we don't word our reports to pre-empt predictable controversies on Wilders. And as you can see, it would not work anyway . We include such details to ensure that everyone knows when a test is and is not sponsored. We try to be as transparent as possible.

Sometimes a customer will ask us to perform a test. This often happens at certain times of the year, such as in the summer when new products are launched. If we agree that the products in the list are fairly comparable we will consider running such tests. These, which are commissioned by the customer, are what we call ‘sponsored’ tests.

The regular quarterly testing is financed by subscribers, which we refer to as partner vendors. These companies support the test by paying a quarterly subscription fee. In return they are allowed certain privileges.

For example, if their product performs well then they are able to use our logos to promote their success. They also have an opportunity to review our logs to discover where problems may lie with their products.

Finally they have an opportunity to challenge our findings (for their own products). This means that, should they wish to argue with our findings, they can do so before we publish the report. This is why you sometimes see comments from a vendor in the report itself – perhaps explaining why they believe the product suffered some issues.

Neutralisation definition is vague: “What constitutes a running threat ? A malicious script that is terminated/blocked as soon as it starts, or some malware that is already active on the computer (e.g. running trojan)?”

A neutralisation occurs when a threat runs on the system but is unable to effect a significant change. As we log every process that runs on the test systems we know if a malicious process ran or not.

A script or Trojan that executes, but which is terminated soon after it starts, sounds like a classic example of a neutralisation in our test.

There is also a further granularity to this type of result. We have the possibility of partial or full remediation.

So if a script ran, was terminated and no changes were made to the system then that would be a neutralisation with full remediation. This means extra points.

However, imagine that a Trojan runs and sets itself up to survive a reboot using a Registry Run entry. The AV product detects the Trojan and removes it, but leaves the Registry entry in place. Or removes the Registry entry but leaves the Trojan file on the disk. That would be a neutralisation without full remediation.

The testing platform is outdated (Win XP) and runs vulnerable software. Hopes that Dennis will switch to Windows 7.

As we noted in the report, there are a lot of people still running Windows XP. According to the research that we’ve seen, just less than half of all Windows users are using XP.

As XP’s life is coming to an end, and Windows 7 has recently become dominant, we will switch to Windows 7 before the end of this year. You can expect one more XP-based report (2013 Q2) before we upgrade.

Regarding the use of vulnerable software, we pre-install it to give the security software a chance to protect the system. To use an analogy: if we were testing car tyre safety we would not do so in optimum conditions. We'd have windy, wet roads. Similarly we use systems that are vulnerable to attack so we can judge how well the security software protects them.

Who are the partner vendors?

As I mention above, we work with all of the main AV vendors but I am not going to name the specific companies that form the partner vendor line-up.

In some cases you can easily tell, because you’ll see our logos on their marketing material. But in other cases either they may choose not to show the logos or, more likely, they may not achieve an award at all.

Desire for paid AVG.

We have tested the paid version of AVG in two earlier tests. There was a strong desire from readers to see free products and, as the paid-for and free versions of AVG *should* receive similar results it seemed sensible to test the free version only.

Why do I think that the free and paid-for versions should receive similar results? Because we are not currently testing email threats or sending malware and exploits over IM. Ours is a web-based test.

Default settings used?

With consumer products default settings are used. However, we disable each product’s ability to report its statistics and other telemetry home to the vendor. Also, we may increase the logging level if that helps us discover issues with the product.

In some cases we may extend the number of TCP ports that the product is allowed to scan for HTTP-based threats. This is because we use a range of ports when replaying the threat sessions.

More programs should have been tested

This consumer test includes nine very well-known products. At the same time we ran a small business and enterprise test, bringing the total number of products to 19.

This involves a vast amount of work and, while we could possibly squeeze a few more products in, it’s not a trivial matter. We hope that as more vendors support the test we'll be able to extend the number of products tested.

Adjusted testing procedures to give benefit to skew the test in favour of one or more products/vendors

We do not adjust our methodology in favour of any vendor or product. We will make small configuration changes if a vendor notes that its product is behaving differently in our environment than it would in the real world.

Our test setup is very close to being the same as many home and business users, but there are always compromises we have to take in order to expose the products to the same threats in the same way. Malicious sites don’t like being tested!

So when we encounter issues we address them to remain fair to all.

DTL tests are on a par with basic magazine tests and are not comparable with AV test labs’ work

Dennis Technology Labs is a testing business that resides within the media company called Dennis Publishing Ltd. We provide test data to magazines within Dennis and to magazines published by other companies.

We test anti-malware products constantly, using state of the art techniques and equipment. I don’t know of any magazine that would be able to summon resources even vaguely resembling the way we work.

As I mentioned above, we are focussed on realistic testing and ensuring that all results are accurately verified and recorded. We don’t run on-demand scans on millions of files like some testers do. We simply take a different approach.

That said, our results are not that different in terms of rankings, to those produced by the other major testers. This is pretty interesting considering our different testing approaches.

No regulation of testing groups

There is no regulation of anti-malware testing, just as there is no regulation of many industries. There is the Anti-Malware Testing Standards Organization, though. AMTSO brings together vendors and testers into a forum for (usually very robust!) discussion.

Dennis Technology Labs is proud to be a member of AMTSO, alongside almost all of the other major testing groups.

From my own, personal point of view, one of AMTSO’s great strengths is that it enables discussion between vendors, testers and anyone else interested enough to engage. It’s not a regulator but it is very influential.

---

I hope that the above answers at least some of your questions.

You can find some more information in various posts on my (personal) blog and I am also on Twitter (@spgedwards) for those who want to follow discussions there.

Many thanks for your time.
Simon Edwards

 

Hi Simon,

thanks for your post. Only one note from my side:

There is only one lab exception (AV-C) which uses more test cases for those real-world tests (about 10x more).

see you next week,
andreas

 

Hi Andreas!
Of course, in your real-world test you use many more URLs. I was thinking only of the drive-by threats that we use.
Looking forward to seeing you very soon!
Si

 

lol.
I believe in one thing that is universal: money talks.
simon's long posting #43 reminds me the meaning of "no trousers, all mouth". sorry to say this but that's my feeling.

 

And you criticised me for stating folk have pre-conceived view??think you have more than proved this for me!

 

To Simon and whoever it may concern,

I vaguely recall Dennis Labs being discussed in past threads on Wilders. Not in a favorable light. I don't mean to be accusatory, but that's part of my frame of reference.

If a vendor doesn't like the results, can he choose not to have the results published ? For av-comparatives, opting out of disappointing tests is (for most tests) not an option.

I was never trying to suggest that you deliberately and consciously would try to skew the results in favor of one particular vendor.

Unsponsored ?

Customers asking for tests, 'These, which are commissioned by the customer, are what we call ‘sponsored’ tests.'

'The regular quarterly testing is financed by subscribers, which we refer to as partner vendors. These companies support the test by paying a quarterly subscription fee. In return they are allowed certain privileges.'

Money talks. Yes, vendors also have to pay av-comparatives, but it is relatively independent.

'Neutralisation definition is vague: “What constitutes a running threat ? A malicious script that is terminated/blocked as soon as it starts, or some malware that is already active on the computer (e.g. running trojan)?”'
A quick note: a malicious script may be neutralized before it has a chance to cause any damage. But, even if the AV doesn't stop the script it doesn't mean that it will cause harm. If the script attempts to exploit a vulnerability on the testing system that simply isn't there, was it ever a threat/was the AV helpful ?

 

OK you can call common sense such as "money talks" as "pre-conceived view", it's fine.
There are facts in this world. It's there, it works that specific way most of the times, no matter you acknowledge it or not.
If you can not distinguish an axiom from an individual view, then you need to study more on that to learn its difference. What it really means is, it's not just some portion of the population that believe it's true, it's well acknowledged truth, believed to be true by majority of the whole population. It's self-evident. You have to admit the earth is round after all, it does not matter what you view is.

 

The same was true for AV-Test and virtually any testing organization (except VB100 and CheckVir, ICSA, etc. whose testing was limited at best in the early days). The industry will give a newcomer low credibility at first. I have to question your intentions in trying to dig up all this.....

I understand that a lot of people put too much stock into test results; but IMO there have been a lot of brickbats thrown at AV-C lately. None of these organizations are a joke; and especially not AV-C given it's association with the University of Innsbruck (do you think that if your tests are not scientifically valid, a well-known university would agree to work with you on a long-term basis?)