DTL Home Anti-Virus Protection Annual Report 2015

I dont know about the protection test, but the performance one reflects my (subjetive) experience.

 

True. I misinterpreted the first sentence of that section.

 

Same here--which is pretty much the opposite of what AV-Marketers AV-Test/AV-Comparatives rate.

The only product that's consistent between all testors and my experience is Kaspersky.

 

Another reason to be suspicious of this latest report from Dennis Labs: Although the report claims that "this test was unsponsored," it also states that this test "collates data" from tests that were "conducted regularly during the previous year (2014)." Weren't some of those tests sponsored by Symantec? For example, the Dennis Labs test report from 16 September 2014 states: "This test was sponsored by Symantec."

What fraction of Dennis Labs' tests are sponsored by Symantec? Does Symantec dictate the methodology used by Dennis Labs during the tests it sponsors?

 

This was discussed in detail in this thread: https://www.wilderssecurity.com/threads/dennis-technology-labs-january-march-2013-tests.346560/ . Dennis Labs is part of U.K. based Dennis Publishing: http://www.dennis.co.uk/

Noting that Symantec products always get "stellar" reviews from PC Magazine, I strongly suspect the same relationship holds true with Dennis. No commercial concern in their right mind is going to portray a major ad revenue client in a negative light lest the client decides to find another source for their ads.

This subsidiary relationship status of Dennis Labs also calls in to question just how "independent" their security product reviews are. Stick with the A-V Comparatives, NSS Labs, etc. reviews if you want an unbiased one.

 

I have a question
Just who has to say it, or report it, for it to be true when it comes to validity and accuracy in malware testing.
Me, I pretty much think all testing organizations do a pretty good job and as such, I think most are right at what they report on. Pretty much the ones that are on or near the top, are the ones folks use here.

But I guess its human nature to flog the messenger, if ours isn't the one with the blue ribbon in their testing.

 

Or that Norton actually is a stellar product (which has been my experience after they had gotten their act together several years ago). I continue to recommend running Norton to anyone who's ISPs provide that product for free. (Mine provides McAfee )

Why on earth would you think AV-* are unbiased? They get paid directly by those whom they are testing and they request the product under test directly from those whom they are testing prior to the test and PC Magazine only uses AV-* in their "reviews". There's a wide path for bias not to mention improper testing methodologies.

 

Fully agree with you Rolo42

Kaspersky is a very solid and consistent antivirus solution in many tests along many years and Norton is quite there too.

Nowdays I prefer my subjective experience than some AV test, but I like to read those reports anyway.

 

There are a couple of things that can be verified for the lab. Are they ISO certified? A-V Comparatives and NSS labs are. How transparent are they on their funding sources. Here is A-V Comparatives disclosure: http://www.av-comparatives.org/funding/ . Does the lab perform and disclose whether the test is sponsored (paid for) or unsponsored? NSS Labs does.

Here is info on other labs:

https://www.av-test.org/en/about-the-institute/partners/
https://www.virusbtn.com/about/advboard

As far as I am aware of, NSS Labs and A-V Comparatives are the only organizations that are ISO certified in security testing procedures.

 

Until we have all the testing organizations presenting prevalence based testing, then each and every report that attempt to do any ranking are completely worthless.

I have voiced my opinion on the subject before and continue to do so.

The current state of testing from all the organizations where they use a collection of samples where a huge portion of samples are practically not hitting anybody, are worthless and misguiding.

This is a general problem with all the labs/organizations, and not specific to the lab who are presenting a report in this thread.

 

DTL is among the AMTSO members.
http://www.dennistechnologylabs.com/services/anti-malware/

Just 'My two cents' (2¢)...

 

I agree. The problem here is who decides which malware is prevalent and which not? AV companies being tested? Also some malware families are more prevalent in some areas in world and other somewhere else. Also some target gamers, other social media users and another P2P users. IMO it's not that easy to use all this variables in testing.

 

Does anyone remember the DTL report where the free version of MBAM scored higher than MSE for protection? I still don't know how that is possible.

 

They are ISO 9001:2008 certified which is a generic quality management certification that's actually been replaced by the 2015 version and has zero to do with AV-testing and it doesn't measure competence. It certified that the company doesn't break any federal laws and it has a quality improvement program. Anybody can garner ISO 9001 certification--including the dealer that totally borked my car last week.

 

True. But it's better than nothing.

The problem with the IT security industry is that it is basically unregulated. As such, there are no standards and oversight agencies to monitor conformance to those standards. And most importantly, a means to penalize non-conformance.

 

For what it's worth, NSS Labs set the testing benchmark a lot higher by their move to "real world" testing methods last spring as noted here: http://bits.blogs.nytimes.com/2015/...-will-hold-security-vendors-accountable/?_r=0 . Their recently completed consumer security product exploit tests utilized such a test platform.

The latest A-V Comparatives real world comparative also used a similar methodology although not as extensive in scope as that used by NSS Labs.

 

Information security is regulated about enough that it needs to be (primarily the public sector and commerce). There are ISO 27000 standards for IT security specifically and PCI DSS for commerce.

What "oversight agencies" exactly do you want and what do you want them to do? The same government who keeps losing PII or the government that wants to ban encryption and privacy?

 

Have you read the DTL methodology? It seems very similar to the AV-Comparatives one to me.

 

Because not all the malware is introduced to the system via an exploit.

 

This DTL report, sponsored by Microsoft, suggests Windows updates are far more important. I guess though that it is only the case if you use Internet Explorer.

Windows Updates vs. Web Threats - dennistechnologylabs.com/reports/s/a-m/microsoft/DTL_2014_Updates_MS.1.1a.pdf

 

Except for one major difference. A-V Comparative provides details on the malware samples used; number and origin source.

 

DTL provide the number. What do you mean by source? The only thing I can find that you might mean is this.

 

I was referring to the actual number of samples used e.g. "450 malware samples were used", etc.. A-V Comparatives and NSS Labs for example always notes that in their reports.

 

DTL notes on their reports "The sample set comprised 100 actively-malicious URLs and 100 legitimate applications and URLs."

 

Ahh, must of missed that. Does sound to me a bit sparse. Just a bit over what the u-Tube jockeys use ..........................