Virus Bulletin August 2011 comparative anti-virus test

Foitinet has been getting better in dection and in gui

 

What?

 

That is odd

 

http://www.virusbtn.com/vb100/archive/compare?tab=onDemand&id=14&id2=1&id3=4&id4=52&id5=&id6=

 

Ya'all realize these are static file tests?

This isn't real-world or "whole securty" tests like av-test.org or av-comparatives run. This is put a bunch of files in a folder and run the scanner. This doesn't test the full security stacks of these products.

VB really should enter the 21st century and stop with zoo only testing - or with "certifications" allowing vendors to claim 100% detection - when we all know that no products detect 100% of threats.

 

raven211, look a little closer at the stats you quote.

Yes, Symantec had 7 fails - but only one fail since 1999 and 55 passes - see the vendor history - http://www.virusbtn.com/vb100/archive/vendor?id=4

Microsoft Security Essentials had 4 passes and one fail

Forefront has had 14 passes

In any case I still don't think the VB test is much good - it doesn't test real time protection, download protection, host IPS, network IPS, source reputation . . . .

We at Symantec found that approx 50% of our detections last year were by IPS.

 

"VB really should stop with zoo only testing"

sorry, but for obtaining the VB100 award, real threats with the highest priority and affecting users are used, not zoo samples.

"...when we all know that no products detect 100% of threats."

youre right, but a 100% Detection rate in this important group is expected by a product.

"it doesn't test real time protection"
this is not true, on-access detection must be guaranteed, too in order to obtain the award.

Only some vendors have reliable detection in this test along years of evaluations, including Symantec

 

after few years, I think Eset is the strongest VB's competitor

 

Outdated testing methodology is outdated.

Your antivirus can have all the detection rate % in the world, but if I can find a virus it doesn't detect in under five minutes and it does nothing to stop me from launching it, its detection rate becomes meaningless.

 

exactly

 

Indeed, when it comes to Virus Bulletin, ESET is the most successful AV vendor.
However, many have questioned the Reliability/Importance/Methodology of Virus Bulletin Tests...

 

Mothedology can be found here: -http://www.virusbtn.com/vb100/about/methodology.xml

 

I know, but there are still many criticizers...

 

look more closely to what you wrote, because in all Anti-malware products you will find a undetected threat.

 

That is exactly my point. When products use nothing but signatures to try to protect, they don't detect it and the system is infected. If they have multiple layers, eg behaviour blockers, web shields, etc. Suddenly it becomes more challenging to infect a system.

Most of the infected systems that come into my house are using such products, products that haven't stepped into the modern world of malware. I could list them off, but I don't want to offend users of the products.

It's not very often a user of Avast, Comodo, Norton, Emsisoft, F-Secure, etc. etc. show up at my door with infected PCs. The reason is that these products have realized they can't keep up to date by releasing signatures for everything and have incorporated very strong alternative methods for protecting.

That is why this test is completely meaningless. A detection rate proves nothing. It just proves specific malware collection teams got lucky, and put those threats in a database before the test was conducted.

 

toxinon12345

The point is that VB is just testing scanning static files for malware. This tells you little about how well the security product behaves in a real-world scenario where the machine be being actively attacked. Say where a user accidently lands on an infected site or inserts a USB device and something runs - say obfuscated code within a web page.

So the VB test tells you nothing about how effective each product's packet inspection, browser protection, system integrity, tamper protection or real-time (not on access - but actual monitoring of processes as they run looking for malicious behaviors) technologies work.

VB needs to move away from static file testing to real-world scenarios. This is expensive and labor intensive, but it gives a far better view into the effectiveness of the solution being tested. There is a lot of divergence between VB's test results and those of av-test, av-comparatives or Dennis Labs because the latter test the full product not just the file scanner.

 

You have a point

 

Can you name a product which only uses signatures for protection? Or can you name a product which doesn't use any of those additional layers you named or any other possible additional layer we know?

 

It would be interesting to compare the products in an offline behavioural test.

 

Not so much I think. It would, imao, only give a very limited view on a product's performance.
'Whole product dynamic test', that's my cup of tea.

 

that would be instresting to do.

 

Testing the protection capabilities of layers like HIPS, behaviour blockers, sandboxes, on-access scanners, File/URL blockers will require of "Dynamic tests" AND "Retrospective/Behavioural tests".

In addition to the web, additional infection vectors could be used such as removable media, network, e-mail, etc.

 

Haha, I just go into assumptions and conclusions too quickly.

 

Now we're talking!

 

That would be a complete AV test!