Threatfire Incremental Detection

If you go over to the Threatfire site there is a bar graph showing the incremental detection Threatfire gives to several AV programs. Most significant are that Trend, Symantec and McAfee were tested, representing over 80% of the paid AV market.

My view is the sample set for this test consists of malware detcterd by Threatfire and missed by signature AV's. It is likely that Threatfire users submit malware that was missed by Threatfire and detected by a signature AV, but that would likely be excluded. In some cases the exclusion may be justified as older forms of malware may be adequately addressed by signature AV's. I also suspect there has been no analysis to determine if any samples are Threatfire false alarms as this is labor intensive.

What I don't understand is how little additional benefit McAfee obtains from Threatfire as I was not aware that McAfee used behavioral detection. Furthermore, Symantec usually tests a little better than McAfee on known malware. AV-Comparatives shows the two to be roughly equal in proactive detection. The results for Trend Micro do not amaze me as an AV that misses 12% has 6 times more missed samples than one that misses 2%, and missed samples are what this is about.

Without attempting to reach a conclusion of A is better than B, can anyone offer an explanation for the results between Symantec and McAfee?

The only conclusion I can safely reach is McAfee does better on proactive detection of the type Threatfire is capable of, but there may be other classes of Malware Threatfire misses, and the performance of either AV on those items is unknown.

 

Even as a ThreatFire fan myself, Ive reached the conclusion that the graph is the same as the one on Prevx's website. Either it's outdated, uses a sample size with enough restrictions to make it realistically meaningless, complete bullshit, or a combination of all three.

 

Makes me wonder if Threatfire would even be needed with some AV's like Avira PP or NOD that are known to have outstanding Heuristics for Real Time Detection.

 

FWIW, in the last week or so I had 1 detection by AntiVirPEC and 4 pop-ups from ThreatFire, 2 expected-2 not. So for me, TF still serves its purpose of helping out AV. (This many detections are rare for me)

 

Well, the restriction on the samples is that the samples were detected by Threatfire in one case and Prevx in the other, and probably both exclude anything missed by Threatfire or Prevx and found by a signature AV. If either Threatfire or Prevx caught everything then we would not need signature based AV's at all.

Differences between the two sets of graphs should be explainable by differences in how each product works.

 

I use ThreatFire aswell, nice behaviour blocker and a good addition to the Avira scan-based heuristics. Low false positive rate, though I managed to produce a few.

But then, I also have KAV, NOD32, F-PROT, AVG, AVAST and BD installed. Oh and also Process Guard.

 

Unfortunately it is a natural reaction for people to immediately disregard such graphs without looking at the proof or data behind them.

Threatfire's I cannot comment on, as it's simply an image that I haven't seen updated in the last 6-12 months.

However, people are actually free to register for Prevx's detection analysis (http://www.prevx.com/register.asp), which if you're accepted, gives you all the detail of all filenames, MD5s, and which ones are caught by one vendor compared to another. Screenshots of the inside section for this area can be briefly seen in the "Prevx is Incremental" powerpoint slideshow available from the same page mentioned previously.

I don't think Threatfire or Prevx are claiming to see "everything" - anyone who did would be stupid to do so. What I think they're both trying to point out is that the big vendors are simply not detecting a LOT of malware samples each and every day - therefore adding their solutions to your security setup would be additive.

 

.
Well said

 

Sorry, the above quote was originally posted by Montpellier

 

Unfortunately it is also a natural reaction for some people to immediately take such graphs at face value when they have been fed with what they are told are insider data about the said graphs.

Filenames and MD5 values of the caught samples? Yes, I imagine that must be very insightful.

The only part of your post that made sense was

Other than that, please realize that the sample set was designed from the get go to make Prevx look good. Samples THEY receive are tested against other vendors: that's it, full stop. What does this say about Prevx's performance against the samples OTHER vendors receive? What does this say about Prevx's detection against a sample set as big as possible so as to give an idea about its overall performance? That's right; absolutely nothing.

 

Hi, folks:

That graph in question has been there since CyberHawk' era, and there are no true data to substantiate it up to this day. I would regard it as advertising banner, no scientific value, needing no further attention. As long as TF works to our belief.

 

Along the same lines, TV commercials dress-up sincere looking actors in white jackets, hang a stethoscope around their necks, & have them give sales spiels for the latest snake oil prescriptions.

By the way (to reveal my ignorance even further) -- what the heck is "incremental detection"?

 

Hi solcroft,

I almost agree with you completely. You're correct that the graph can't say anything about Prevx performance against the samples recieved by other vendors - they don't publish their stats so we'll never know. You're absolutely right that it says nothing about Prevx's detection against a very large sample set either But we're not claiming that it does.

This graph does one simple thing. It shows how those AV vendors peformed when tested against the samples that we have received from our own users in the last 24 hours - all the samples we received! This graph is not based on some "designed" set of samples to make us look good - it's based on every sample we receive.

All the data behind the stats is available to those that register.

More information on this can be found in our powerpoint presentation on Incremental Protection - http://info.prevx.com/download.asp?GRAB=FRONTPPS.

The tests and the publishing of their results are published daily.

It's also worth pointing out that we're saying nothing about how well these vendors will perform with the same sample set 7, 14 or 28 days later. Most by then will detect them we're sure. That is why we say we offer Incremental Protection - we fill the gap while your AV vendor catches up.

Prevx

 

Therein lies the problem.

I assume the graph is there for marketing purposes - it is intended to imply that other vendors miss a large percentage of malware on a daily basis, while Prevx catches them. How far is this true? Does your sample set really represent the total amount of malware in circulation that day? How do you know that, while vendor A misses x% of the malware Prevx detects at any given time, Prevx is not missing 2x% or 3x% of the malware that vendor A detects? But of course, that's not what the graph is intended to show.

The average Joe Schmoe will, upon seeing the graph, be given the impression that Prevx is exponentially more effective than other vendors at catching malware. Given that Prevx includes behavioral detection technologies, I am inclined to believe that this is indeed true. However, it's not something that can be logically deduced from the graph unless one jumps to conclusions. The sample set used, as you say, are samples that Prevx received and already detects. You seem to be saying that I was implying Prevx doctored the set to produce the pretty graphs on their front page; I never said that, and that's not the case anyway. When you only test using a sample set you already score 100% on, you don't have to doctor the set in any way - it's already pre-selected to favor Prevx in the first place.

Out of curiosity, will Prevx be including Avira and Kaspersky in the graphs anytime soon? I'm personally quite interested in seeing the results, actually.

 

We agree that is a problem. But until other vendors publish their stats we will never know. It's fairly obvious to anybody that other vendors will be finding samples that we don't detect - afterall, we can only detect what we have seen from the Prevx community. Can a few hundred thousand active agents see everything? Nope. But the more agents we get the more we see.

All we're saying is that given a set of malware they was first seen today we seem to be doing better than the AV vendors we've tested against. Just because we see a malware sample for the first time today doesn't mean that it's actually new today. Given we have a small community of agents , it's far more likely that the sample has been in the wild for days or even weeks. Given that fact it's quite sad that a large percentage of the samples aren't detected by the AV vendors. Of course everybody is free to make their own conclusions. And we are of course happy to work with any AV vendor to help them improve their reaction times.

My apologies if I read your post incorrectly. I worded my reply this way as you refered to the tests being designed. I was simply trying to indicate that there is no design element, we publish everything warts and all. We look forward to a vendor reaching 90% or more as it will add even more strength to our message.

Personally, I hope so. I hope we can get a much larger number of AV vendors included. Whether we can depends on a number of things - some of which are out of our control.

Darren

 

Yep.

Correction: First seen today by Prevx. And yes, given this sample set, Prevx does better than the competition. Which is my whole point; it's fairly obvious the graph is meant to convince the average Joe Schmoe that Prevx is vastly superior to the other products shown, but unless a leap of logic is involved, that's actually not what the graph shows at all, due to the biased sample set used.

 

Sorry bellgamin, nobody seems to want to explain.
By the way, I've seen some good products that have some really bad commercials, but that only convinces me that the commercial is bad-not the product. But then again, I'm not smart enough to know what "incremental detection" is either...

 

No No, you might be a AV expert, but you really really need to keep up to date with the innovative security solutions out there today.

Though opions differ on what specific products to use, it is generally understood you should have a setup like below (or you can replace them with alternative products that do the same thing).

1)Antivir - Antivirus
2)Boclean - Antispyware
3)Threatfire - "Behaviorial anti-malware"
4)GesWall - Sandbox
5)Snoopfree - antikeylogger
6)Eqsecure - HIPS "classical"
7)Comodo Memoryguard - Buffer overflow protection
Comodo firewall v3 - Software firewall
9) McAffee siteadvisor - warns of malicious sites
10 Retunril virtual system - "Shadow" virtual system

These 8 products represent 10 different and distinct layers that everyone should have. They can be used because they do not overlap in function and uses different approaches to preventing malware and hence each has a role to play.

Add the standard defenses like hosts file, iespyad, spywareblaster, running IE with dropmyrights and running Firefox with no-script/adblock etc, hardening your system with xpsecure, importing ips to blacklist and block with peer guardian etc...

This is way better than running all antiviruses and one outdated Processguard

 

Some people prefer a "minimalistic" approach and somehow manage to stay malware-free.

 

Yes, let's hope Lusher's post was meant ironically

 

Incremental detection? I would assume, that it does something the same as what an incremental backup does. Only backing up, what's changed/added. Detecting incrementally could mean: looking for the newly added/changed stuff, as it remembers what's been scanned already.
This is a pure personal approach and if someone knows exactly......I'd be happy to hear.

 

Hi,

Your post reminds me of a very old event.. back to my teens era.
During art /painting class, a question was popped up " Let mix all those fancy colours, red, orange, purple... and on and on. What colour will this process result in ?" We were all silent suddenly, until someone said " BLACK". A very fundamental and basic shade !

 

I was under the impression that if you mixed any number of different colors, you'd always end up with **** brown?

Quite apt don't you think!

 

Yes, it,s needed for sure.

 

I don't know why you are so sure. Perhaps you can explain why. If McAfee, which is good but not super good, caught 85% of the select sample, it makes me wonder what some of the other higher scoring (on published tests) AV's might do. Then again, Symantec usually beats NcAfee in the same tests, but it only found about 55% of the sample.