[Thread split]MRG Flash Tests 2012

Yes but it is not easy. You have no choice but to look at past generations to predict what they are likely to do to break free from the security collective and then work around these changes ahead of time. If you are dealing with a particularly nasty infection you may even need to make more than 1 prediction just to be sure. Even then this is sometimes not enough, sometimes you have to start over because the changes are just too great. In these cases being fast is the only option (or to block the actual source).

 

Me too! As long as I don't get bitten ...by the malware bug

 

23.07.2012 zero hour plus six hours later.

 

I'm really loving those small test threads.
Kinda interesting.

Thanks for the update malexous!

 

Interesting that Panda failed the same 3 after 6 hours, cloud an all I mean.

I think I read in the Prevx (WSA) (cloud and all) forum that they did react minutes after, when missed a detection.
I really miss WSA in these tests.


/E

 

All i can say about Avast is well done and congratulations to the team.

 

Twelve hours later.

 

Is it just me or does MRG samples 0 hour samples always seem well detected(even if missed by some) yet what Nosirrah is tracking is the exact opposite.

Regardless of whoever the vendor is they all seem to be struggling so would MRG not provide a more accurate picture if they used the same source ?

Unfortunetly and excuse the cynic in me but have already come to the conclusion that MRG will not publish data that would be consistently damning of any of their paying clients.They soon would have no clients afterall

That said it would still be interesting to see one of their detection tables based around Nosirrahs test findings..predominently fails across the board and that includes vendors that are paying clients to MRG services with near perfect scores todate in their flash tests...

 

You should not compare VirusTotal with MRG Effitas Flash Tests.

AV Comparative Analyses, Marketing, and VirusTotal: A Bad Combination

Scroll down for MRG Effitas Flash Tests methodology: http://www.mrg-effitas.com/current-tests/flash-test-results/.

 

@m0use0ver

You have to keep in mind that me and MRG are trying to demonstrate two totally different points. MRG is picking a handful of nasty but very common infections from different sources every test and I have cherry picked 2 sources I know that everyone will be terrible against.

MRG is trying to show that when you remove the element of being able to prepare for an attack it is much more difficult to pass their test. Keep in mind that MRG is also including blocking the source itself, I am not at all taking this into consideration.

My goal is a lot more simple. I am simply trying to demonstrate that "current malware" needs to be redefined to a much smaller window of time. I list the full information for a reason as well, I want the vendors to get these samples and then attempt to prepare for the coming mutations. The sources for these infections are also known. I have posted them directly here and on our forums and anyone paying attention should already have them. Allowing everyone to prepare is by design emphasizing the terribly small window of relevance when it comes to "current malware".

 

If the sources are known, why aren't the vendors who have missed detections (in the short timeframe as per your test) not adding signatures much quicker than it would seem?

 

I cannot answer that. I would like to reiterate that I selected 2 sources that I knew ahead of time would have terrible detection. One is an exploit drop and the other is a fake scanner drop. These types of infections are well known to have poor detection and a very rapid morph rate.

 

Comment by Sveta.

 

To MRG- You also may want to (for your amusement) note the time of day as well as the day of the week that the malware was initially released. The number of people and the amount of resources allocated to the emergency response teams vary depending on the company's size and true focus.

It can be seen that response will differ with respect to the time parameters listed above- basically great if zero hour is 0900 local time on a weekday, but not so good at 0100 local on a Sunday.

 

That's great that DefenseWall is there! I love to see how it beats all malware.
Why no Comodo?
Why no AppGuard?
They definitely must be there, please.

 

MRG and Comodo don't like each other.

 

Nonsense. Hashes are for transparency not for VT-rechecking.
f.e.: They claim x misses y and call y "Zeus-variant". Wow, a "believe it or not" game...

And because they use only a few samples posting hashes will not be the big problem.

 

do you get also hashes from av-comp and others, do not think so, can not see the problem.

 

First: This are often large tests with more samples.
Second: Because others are not transparent it doesn't mean that all have to do that way.
Third: Your example AV-C doesn't have such a strange "history" like MRG (SSUpdater "we not" lies, different names here on wilders, Comodo story...). So transparency is a must for MRG.

 

That makes their case worse, not better. No matter what its 'generate hash, add to list next.....'. They don't disclose the information that would make it obviously why so many vendors did better than 85%.

In Sveta's case live sources can be blocked at the source so the actual samples are meaningless. Sources die far too quickly to do a test and report them in a way that can be verified. He could make it sample based only but then it becomes a fake test and completely invalid.

There are list of live sources all over the web and very simple ways to setup your own test. Anyone that wants to verify this new way of testing can any time of any day. If anything this is a lot MORE disclosure as anyone can replicate the test with a little work.

You should in fact NEVER trust a test that you cannot replicate in any meaningful way.

For the record, I have received every sample we have ever missed in his testing without issue. Also, if you were to give 100 samples to Sveta that were collected 3 days ago he would NOT put them into a test because he knows that they are already obsolete.

 

Really? That's a pity.
Could you give a link to the source of this info, please.

 

They had some kind of misunderstanding in the past. I forgot why.

 

Hopefully they will get along.
Imho UppGuard deserves MRG attention as well.

 

I never said they are "better".

His test is not only sources based, also family based cause he reports malware sample/family names.

And - at last for me - such type of tests would be more interesting if there would be
a.) details, what blocked the file (signature, proactive detection etc.), even if it is not in all cases clear to find out.
b.) transparency, which is for me more than posting a few tables