Latest test from the Malware Research Group Project #19

A reason I won't be using it if it doesn't change.

 

Oh well, some people care about high detection rates, some care about false positives, my opinion is: I can deal with FP's, but I hate the situation when I have to remove malware manually because my AV missed to detect it .
My theory is , and I'm not defending a-squared now, you can solve the problems with false positives easily (if you listen to complains by users), but it is not as easy to improve detection rate without producing a few FP's if you are using other methods of detection (heuristic....), and of course the less FP's the better, and the higher detection rate that goes with that the better

@Pleonasm

In my opinion, on demand scans are the best way of testing for now, many talk, few invent new methods.
In theory what AV detects on demand, it should be able to detect in real time, therefore block, I don't want to go into many details now, but imagine how much fun would it be to test AV's in real time with ,lets say, 100k of samples, mission impossible

 

The logic is true, if and only if the AV has the signature used for detection at the time that the threat occurred on the user’s PC. Since there is a delay between the time that a new malware instance becomes “live” in the wild and the identification of the threat by the AV vendor and the issuance of a signature update to the product’s users, a detection rate for an on-demand test used with historical malware samples may not necessarily translate into the performance of the product for dynamic detection in real-time.

Retrospective on-demand scan testing (in which the AV signatures predate the malware samples) is a step in the right direction, but still falls shorts of the realism of dynamic testing, in my opinion.

Difficult, yes; impossible, no. There is hope on the horizon. For example, AV-Comparatives states that dynamic tests “will be published later this year” (see here).

 

That is the first test of OA++, was looking for any other reviews... and found nothing. Also interesting is that MRG team is opening up their own forums

 

I rather have both, so that's what I use.

 

It is very hard to have both, the more advanced methods of detection (aggressive), the more likely it will be that FP's may occur

 

OA++ is in pre-release, not really any reviews out there yet - but I'd be happy to have it tested by independent testers, and don't mind them checking out the beta.

 

Any news on when the final release is coming out?

 

Was hoping to do it today, but I have a strange issue on my machine that doesnt reproduce that needs to be checked

 

Good point - but I personally think I've that in Norton 2010.

 

I don't understand how can you take this tests seriously and that especially. 600k samples. Where did they get them? Are they sure all of this is a real working malware? Did they test every sample? No one could do that.

The problem is that such "number one" products will bring you tens of FP, but can't safe you from really dangerous malware and couldn't cure your system even if they add a signature later.

That's I am talking about.

That's why I respect them and use their products.
In one of the interviews Eugene Kaspersky said something like: "I can make anivirus that will pass all the tests but it will not protect customers properly".

 

Hello and welcome to Wilders.

I suggest you ask your questions by Personal Message (PM) to guest. He is well informed about those tests and may even be one of the proponents.

As for me, my rule is this...

A- If my favorite antivirus application did well on the test, it's an excellent test.

ON THE OTHER HAND

B- If my favorite antivirus application did NOT do well on the test, it's a
lousy test.

 

BG, your post made my day, thank you.

 

I agree with Bellgamin completely, but only if his favourite application is Online Armor, otherwise his logic is flawed.

Thanks for a great post Bellgamin, that was a fantastic grin to start the day.

 

I find this statement illogical. If he knows how to protect customers properly and how to pass all the tests, why not to do the both ?

If AV fails to detect some malware in a test, it will surely fail to detect it in a real life as well.

 

OA & Ikarus -- a marriage made in heaven.

 

Ditto!

 

Thanks

Rule for kid, not for sensible person you should be.

There is nothing illogical.

Some virus analitics confess that it's easier to add crap to the bases instead of listen to customers exclamations about bad percentages in the "tests". So they do that and the bases grow by the "in-the-wild" "malware" independent "testers" uses. I saw some samples from such collections. Every fith file is a garbage.

You misunderstood me, may be because of my english? If only it was malware! Not crap collected and added by hash. I can tell you my thoughts about Ikarus "anti-virus", which engine A-squared uses, but I don't want to listen it users screams. I can only say that then I send one of Kido\Conficker\Downdap\Shadow modifications to VirusTotal, Ikarus was the one only from 41 who can't recognise it.

 

What do you mean when you say "it is garbage" ?

Does a way a crap is detected (hash, heuristic, sig) do any difference to an end-user ? Do you know any reliable way to test AV detection rate ?

 

@Nomad Soul

I'm not quite sure what is your point here, are you saying that testers are using "crap" in their tests, if so how come most of the programs detect over 97% of such samples?

False Positives are problematic for beginners, but always a good excuse for some You always have a choice to use what suites your needs the most, if that is Dr.Web, so be it

As I got to know a few of the people you are accusing of adding "crap" to their tests, I will quote one of them

"... and in real world conditions there are thousands of new samples being discovered every day.
.......1/100 of those samples is truly malicious....
.....indeed we are sharing all the files we get as that is what they requested , what they add to their databases it none of our concern....
......questioning AV tests is very easy if you don't like the results....."

 

Hi, I can see your point to a degree. Some malware samples may be harmless, inflict almost no damage, be pathetic batch files or exploits which are ineffective against up to date systems and for the layman, it would be tricky to differentiate these from the more dangerous or truly malicious ones.

With a huge number of samples, it would not be possible to check every one - obviously not. This said however, any credible test will use a very large number of samples as this will give a more accurate and representative assessment of an AMs capability.

Having a smaller sample size will reduce the validity of the tests and unless the sample was so small that the tester was able to examine every single piece of malware in the sample to determine their true malware status, you will not avoid the problem of having junk files included. sample this small would yeild meaningless results.

If you look at several different tests, from various sites, I think it is true to say there is a pattern of results and the top AMs generaally circulate in the top 1-4 places.

It would be quite simple to test what Dr Web, KAV etc miss and see if they are truly malicious - indeed, i have done this with several AMs and can confirm some of what is missed is junk, but a good proportion is not.

Those AMs that tend to do best in these tests also tend to have fewer truly malicious bits of malware in what they misss.

I would sooner have an occasional FP or something flagged as malware that may only be slightly risky, than have my system infected.

So, whilst it is true to say noone can guarantee any large sample wont contain some junk, I would argue it is ilogical to say you cant take such tests seriously.

Puss

 

I mean damaged malware that doesn't work or other files that couldn't be called malware and isn't harmful.
May be I do not right in formulating, but I mean it is crap or garbage or junk files in malware collection because it isn't malware.

Yes, because it shouldn't be detected.

Can you collect 600k of malware! samples yourself and check all of them without sending to any vendor or VirusTotal-like services? NO! It's the answer to your question too.

I can't say they add crap, I say they couldn't check for real all the samples they collect.

FP and Avira, for example, said they don't edit\solve signatures which detect damaged(nonworking - my comment.) malware.

Sure but it is not good then vendor creates a feel of protection, but don't protect properly.

What do you mean by this? Explain please.

Larger collection of unchecked samples with uncontrollable amount of FP don't make test more validate.

It doesn't matter how many tests if they use hundreds of thousands of "something".

Such tests after test were one of the reasons I chose my anti-virus product. Even in this single topic there is person who noticed the same:

The problem is what "SOME" amount is quite huge.

 

Tests can never be absolutely accurate, but they usually give a pretty good indication of the ups and downs of AVs companies. If you are among the first 4 in terms of results it is pretty good going. I agree with Saraceno that Dr Web is a bit of a puzzle, reading posts from different backgrounds it seems to get contradictory results particularly considering that it is one of the most mentioned tools when cleaning computers.

Is there really a difference between an AV good at detecting (meaning denying access) and one good at cleaning? Perhaps an AV expert can answer that as long as it is an unbiased analyses (hard nowadays to get people to be neutral).

 

Im sorry, but you are wrong.

The size of the sample will not alter the proportion of non malware included.

A larger sample will (statistically) be more representative of the malware that exists as a whole and so the tests will give a better indication of real world performance.

I understand your fundamental point that some vendors may add definitions just for the sake of it - agreed, but, this said, I would argue the AMs that came out top in this and other similar tests have the best detection rates.

I have done quite a bit of testing on new (less than 48 hours old) and limited spread malware - both live infection detection and on demand detection and Dr Web and KAV have NEVER been in the top. A2, Avira & Prevx are always significantly better at catching new maalware than any other AM I test.

As a matter of interest, with some very new malware samples received, there is about 10% which is junk. On a number of occasions, I have had the samples analysed by a (nameless) AM vendor to have the junk stripped out and tested the true malware samples against my usual set of AMs - and the results are aalways much the same as with the mixed sample. KAV and Dr do not do any better.

Puss

Like I said before, it is better to have a few FPs than an infected system