Malware Research Group #23 Test

Uh... I'm starting to be speechless. See - in fact, I don't need ANY signature at all. I simply flag any file as malware. With your test, if I write a batch file which will loop thru a directory w/ your samples and mark every single one as malware, I have detected everything.

 

... and Doktornotor-Batch-AV would be the lightest AV Wow, you should create it soon ...

 

No mate, I'm speechless already. I totally understand what you are saying, but that has nothing to do with this test.
As far as I know, no application uses such technique of detection and I'm pretty sure that none will.

Everybody can sit down and code a batch that will say detect all, but where will that lead you, what will you achieve

You know there is a reason why AV's use signatures, heuristics, clouds....

 

Sigh... even absolutely terrible product with obnoxious % of false positives that will most likely ruin users' OS in minutes can (and probably will) win your test. The batch file was just an extrapolation which shows how absurd your test it. It'd be like a reverse test where the product w/ least false positives would be a winner, no matter how many samples it missed. Download Placebo AV and there's your champion.

 

Ah thanks for enlightening me
Quick question tough, if its that simple, how come all AV's used in this test didn't detect 100%?

 

Well, that's probably because vendors (fortunately) don't care about your tests. A prime example how this flawed methodology stuff can go extremely wrong once vendors get foolish enough to take it seriously - that'd be Matousec.

 

The golden MRG award

 

Well good luck

 

Guys relax, that's why sometimes developers leave this forums
No need for flame war, if you don't like their tests just ignore them.

IMO, i like these tests and for me they still prove something, you may call me Noob . . . Ooops that's my nickname

 

So, you don't care about innocent files getting wiped from your system? It's kinda a problem for others

 

These files are secured now, no malware can infect them

 

Ah... stupid me. How could I have missed such a revolutionary concept!?

 

Hehe!

But hey, let's stop our Malware Experts Group bashing now ok

 

Lets all take a deep breath know, nobody is impressed with whats been said here. If you want to start a flame war, in that I will never participate so don't waist your time.

I believe that I am talking about one thing and you (doktornotor and ance) are talking about something else.
A good AV should be good at many things and I will name a few:

1. Detection
2. Real Time
3. Removal
4. Heuristics
5. Low false positives
6. System resources
and more

You can pretty much do a test on all of the categories listed above and the data from these tests is of great value for developers, and more then you can imagine.

I get a feeling that you think that False Positives are more important then everything else, well they are not. A good AV should have low false positives rate, but also a good detection rate, unfortunately it is very difficult to find a perfect balance between the two.

As a conclusion, detection rate is what we tested this time, next time we will test prevention and after that removal as a good AV should be good in every category of testing

 

I think what doktornoktor tried to say was:

"Principle 4: The effectiveness and performance of anti‐malware products must be measured in a balanced way.
It is difficult – and can be misleading – to summarize product efficacy with a single measurement. Testers are encouraged to present multiple measurements of product performance in different areas in order to allow users to make an informed decision.
For instance, testers should appropriately balance false negative and false positive test cases. A product that is successful at detecting a high percentage of malware but suffers from a high false positive rate, may not be “better” than a solution which catches less malware but which generates less false positives." (source: www.amtso.org)

 

Yeah, exactly that...

 

Well that does not differ much from my philosophy and is pretty much what I said in my previous post.

One thing we like to do is test for each category separately, in few months we will be adding false positives to to our tests but as a separate category.

In my opinion the best thing for a user is to see various tests and then make a choice. Always keep in mind that what works for me may not work for you.
Some people care for high detection, some for low false positives and then there are those with restricted system resources.

 

1. useless static detection test
2. that many samples cannot be verified

if there truly was 250k+ samples, how can any antivirus get 99.8%, what a joke!

i fail to see any use out of this 'test', ONCE AGAIN!

how about i go and search just 10 samples, and they wont detect them all, (not even new ones) so your test means nothing to me and shouldn't to anyone else.

all i can think of, is your downloading large amounts of virus collections that X-Antivirus said were infact samples, and you check these on-demand against multiple products.

i see countless flaws in your results alone, and dont understand where the genius-of-testing-methods is, care to enlighten us all?

if this is what you guys call testing, im afraid your stuck in the past, this is 2010 - Test properly or not at all, its not about Quantity, its about Quality of the test that makes them reputable.

 

Thanks a lot for opening my eyes, now I can sleep better

 

Sarcasm wont work on me sonny jim, maybe you aint aware that us brits invented it

but in all seriousness, maybe it SHOULD open your eyes & stop wasting yours & everyone elses time.

you wouldn't be a Serbian virus collector would you?

 

i dont know to much about this security testing company but to be honest i like to see their videos from youtube ,thanks Sveta MRG and chris also

 

Too bad you didn't invent a bit of civility to go with it.

 

Nice one.

 

Hahaha, we are going off topic

 

There are many tests like this one around the internet. They do have their relevance but it is limited.

It would be fascinating to learn how many of the files are indeed malicious.