Threatfire Incremental Detection

[Diver]

If you go over to the Threatfire site there is a bar graph showing the incremental detection Threatfire gives to several AV programs. Most significant are that Trend, Symantec and McAfee were tested, representing over 80% of the paid AV market.

My view is the sample set for this test consists of malware detcterd by Threatfire and missed by signature AV's. It is likely that Threatfire users submit malware that was missed by Threatfire and detected by a signature AV, but that would likely be excluded. In some cases the exclusion may be justified as older forms of malware may be adequately addressed by signature AV's. I also suspect there has been no analysis to determine if any samples are Threatfire false alarms as this is labor intensive.

What I don't understand is how little additional benefit McAfee obtains from Threatfire as I was not aware that McAfee used behavioral detection. Furthermore, Symantec usually tests a little better than McAfee on known malware. AV-Comparatives shows the two to be roughly equal in proactive detection. The results for Trend Micro do not amaze me as an AV that misses 12% has 6 times more missed samples than one that misses 2%, and missed samples are what this is about.

Without attempting to reach a conclusion of A is better than B, can anyone offer an explanation for the results between Symantec and McAfee?

The only conclusion I can safely reach is McAfee does better on proactive detection of the type Threatfire is capable of, but there may be other classes of Malware Threatfire misses, and the performance of either AV on those items is unknown.

[solcroft]

Even as a ThreatFire fan myself, Ive reached the conclusion that the graph is the same as the one on Prevx's website. Either it's outdated, uses a sample size with enough restrictions to make it realistically meaningless, complete bullshit, or a combination of all three.

[Wordward]

Makes me wonder if Threatfire would even be needed with some AV's like Avira PP or NOD that are known to have outstanding Heuristics for Real Time Detection.

[19monty64]

{QUOTE-> Makes me wonder if Threatfire would even be needed with some AV's like Avira PP or NOD that are known to have outstanding Heuristics for Real Time Detection. <-QUOTE}
FWIW, in the last week or so I had 1 detection by AntiVirPEC and 4 pop-ups :o from ThreatFire, 2 expected-2 not. So for me, TF still serves its purpose of helping out AV. (This many detections are rare for me) ;D

[Diver]

{QUOTE-> Even as a ThreatFire fan myself, Ive reached the conclusion that the graph is the same as the one on Prevx's website. Either it's outdated, uses a sample size with enough restrictions to make it realistically meaningless, complete bullshit, or a combination of all three. <-QUOTE}


Well, the restriction on the samples is that the samples were detected by Threatfire in one case and Prevx in the other, and probably both exclude anything missed by Threatfire or Prevx and found by a signature AV. If either Threatfire or Prevx caught everything then we would not need signature based AV's at all.

Differences between the two sets of graphs should be explainable by differences in how each product works.

[Stefan Kurtzhals]

I use ThreatFire aswell, nice behaviour blocker and a good addition to the Avira scan-based heuristics. Low false positive rate, though I managed to produce a few.

But then, I also have KAV, NOD32, F-PROT, AVG, AVAST and BD installed. Oh and also Process Guard. ;D

[Montpellier]

{QUOTE-> Even as a ThreatFire fan myself, Ive reached the conclusion that the graph is the same as the one on Prevx's website. Either it's outdated, uses a sample size with enough restrictions to make it realistically meaningless, complete bullshit, or a combination of all three. <-QUOTE}
Unfortunately it is a natural reaction for people to immediately disregard such graphs without looking at the proof or data behind them.

Threatfire's I cannot comment on, as it's simply an image that I haven't seen updated in the last 6-12 months.

However, people are actually free to register for Prevx's detection analysis (http://www.prevx.com/register.asp), which if you're accepted, gives you all the detail of all filenames, MD5s, and which ones are caught by one vendor compared to another. Screenshots of the inside section for this area can be briefly seen in the "Prevx is Incremental" powerpoint slideshow available from the same page mentioned previously.

I don't think Threatfire or Prevx are claiming to see "everything" - anyone who did would be stupid to do so. What I think they're both trying to point out is that the big vendors are simply not detecting a LOT of malware samples each and every day - therefore adding their solutions to your security setup would be additive.

[benton4]

{QUOTE-> I don't think Threatfire or Prevx are claiming to see "everything" - anyone who did would be stupid to do so. What I think they're both trying to point out is that the big vendors are simply not detecting a LOT of malware samples each and every day - therefore adding their solutions to your security setup would be additive <-QUOTE}.
Well said

[benton4]

Sorry, the above quote was originally posted by Montpellier

[solcroft]

{QUOTE-> Unfortunately it is a natural reaction for people to immediately disregard such graphs without looking at the proof or data behind them. <-QUOTE}
Unfortunately it is also a natural reaction for some people to immediately take such graphs at face value when they have been fed with what they are told are insider data about the said graphs.

Filenames and MD5 values of the caught samples? Yes, I imagine that must be very insightful. ::)

The only part of your post that made sense was
{QUOTE-> I don't think Threatfire or Prevx are claiming to see "everything" - anyone who did would be stupid to do so. What I think they're both trying to point out is that the big vendors are simply not detecting a LOT of malware samples each and every day - therefore adding their solutions to your security setup would be additive. <-QUOTE}
Other than that, please realize that the sample set was designed from the get go to make Prevx look good. Samples THEY receive are tested against other vendors: that's it, full stop. What does this say about Prevx's performance against the samples OTHER vendors receive? What does this say about Prevx's detection against a sample set as big as possible so as to give an idea about its overall performance? That's right; absolutely nothing.

[Perman]

Hi, folks:

That graph in question has been there since CyberHawk' era, and there are no true data to substantiate it up to this day. I would regard it as advertising banner, no scientific value, needing no further attention. As long as TF works to our belief.

[bellgamin]

{QUOTE-> Unfortunately it is also a natural reaction for some people to immediately take such graphs at face value when they have been fed with what they are told are insider data about the said graphs. <-QUOTE}Along the same lines, TV commercials dress-up sincere looking actors in white jackets, hang a stethoscope around their necks, & have them give sales spiels for the latest snake oil prescriptions. ;)

By the way (to reveal my ignorance even further) -- what the heck is "incremental detection"?:P

[ghiser1]

{QUOTE-> Other than that, please realize that the sample set was designed from the get go to make Prevx look good. Samples THEY receive are tested against other vendors: that's it, full stop. What does this say about Prevx's performance against the samples OTHER vendors receive? What does this say about Prevx's detection against a sample set as big as possible so as to give an idea about its overall performance? That's right; absolutely nothing. <-QUOTE}

Hi solcroft,

I almost agree with you completely. You're correct that the graph can't say anything about Prevx performance against the samples recieved by other vendors - they don't publish their stats so we'll never know. You're absolutely right that it says nothing about Prevx's detection against a very large sample set either But we're not claiming that it does.

This graph does one simple thing. It shows how those AV vendors peformed when tested against the samples that we have received from our own users in the last 24 hours - all the samples we received! This graph is not based on some "designed" set of samples to make us look good - it's based on every sample we receive.

All the data behind the stats is available to those that register.

More information on this can be found in our powerpoint presentation on Incremental Protection - http://info.prevx.com/download.asp?GRAB=FRONTPPS.

The tests and the publishing of their results are published daily.

It's also worth pointing out that we're saying nothing about how well these vendors will perform with the same sample set 7, 14 or 28 days later. Most by then will detect them we're sure. That is why we say we offer Incremental Protection - we fill the gap while your AV vendor catches up.

Prevx

[solcroft]

{QUOTE-> You're correct that the graph can't say anything about Prevx performance against the samples recieved by other vendors - they don't publish their stats so we'll never know.

This graph does one simple thing. It shows how those AV vendors peformed when tested against the samples that we have received from our own users in the last 24 hours - all the samples we received! This graph is not based on some "designed" set of samples to make us look good - it's based on every sample we receive. <-QUOTE}
Therein lies the problem.

I assume the graph is there for marketing purposes - it is intended to imply that other vendors miss a large percentage of malware on a daily basis, while Prevx catches them. How far is this true? Does your sample set really represent the total amount of malware in circulation that day? How do you know that, while vendor A misses x% of the malware Prevx detects at any given time, Prevx is not missing 2x% or 3x% of the malware that vendor A detects? But of course, that's not what the graph is intended to show.

The average Joe Schmoe will, upon seeing the graph, be given the impression that Prevx is exponentially more effective than other vendors at catching malware. Given that Prevx includes behavioral detection technologies, I am inclined to believe that this is indeed true. However, it's not something that can be logically deduced from the graph unless one jumps to conclusions. The sample set used, as you say, are samples that Prevx received and already detects. You seem to be saying that I was implying Prevx doctored the set to produce the pretty graphs on their front page; I never said that, and that's not the case anyway. When you only test using a sample set you already score 100% on, you don't have to doctor the set in any way - it's already pre-selected to favor Prevx in the first place.

{QUOTE-> It's also worth pointing out that we're saying nothing about how well these vendors will perform with the same sample set 7, 14 or 28 days later. Most by then will detect them we're sure. That is why we say we offer Incremental Protection - we fill the gap while your AV vendor catches up. <-QUOTE}
Out of curiosity, will Prevx be including Avira and Kaspersky in the graphs anytime soon? I'm personally quite interested in seeing the results, actually.

[ghiser1]

{QUOTE-> Therein lies the problem.

I assume the graph is there for marketing purposes - it is intended to imply that other vendors miss a large percentage of malware on a daily basis, while Prevx catches them. How far is this true? Does your sample set really represent the total amount of malware in circulation that day? How do you know that, while vendor A misses x% of the malware Prevx detects at any given time, Prevx is not missing 2x% or 3x% of the malware that vendor A detects? But of course, that's not what the graph is intended to show.
<-QUOTE}

We agree that is a problem. But until other vendors publish their stats we will never know. It's fairly obvious to anybody that other vendors will be finding samples that we don't detect - afterall, we can only detect what we have seen from the Prevx community. Can a few hundred thousand active agents see everything? Nope. But the more agents we get the more we see.

{QUOTE->
The average Joe Schmoe will, upon seeing the graph, be given the impression that Prevx is exponentially more effective than other vendors at catching malware. Given that Prevx includes behavioral detection technologies, I am inclined to believe that this is indeed true. However, it's not something that can be logically deduced from the graph unless one jumps to conclusions. The sample set used, as you say, are samples that Prevx received and already detects.
<-QUOTE}

All we're saying is that given a set of malware they was first seen today we seem to be doing better than the AV vendors we've tested against. Just because we see a malware sample for the first time today doesn't mean that it's actually new today. Given we have a small community of agents , it's far more likely that the sample has been in the wild for days or even weeks. Given that fact it's quite sad that a large percentage of the samples aren't detected by the AV vendors. Of course everybody is free to make their own conclusions. And we are of course happy to work with any AV vendor to help them improve their reaction times.

{QUOTE->
You seem to be saying that I was implying Prevx doctored the set to produce the pretty graphs on their front page; I never said that, and that's not the case anyway. When you only test using a sample set you already score 100% on, you don't have to doctor the set in any way - it's already pre-selected to favor Prevx in the first place.
<-QUOTE}
My apologies if I read your post incorrectly. I worded my reply this way as you refered to the tests being designed. I was simply trying to indicate that there is no design element, we publish everything warts and all. We look forward to a vendor reaching 90% or more as it will add even more strength to our message.
{QUOTE->

Out of curiosity, will Prevx be including Avira and Kaspersky in the graphs anytime soon? I'm personally quite interested in seeing the results, actually. <-QUOTE}
Personally, I hope so. I hope we can get a much larger number of AV vendors included. Whether we can depends on a number of things - some of which are out of our control.

Darren

[solcroft]

{QUOTE-> We agree that is a problem. But until other vendors publish their stats we will never know. <-QUOTE}
Yep.

{QUOTE-> All we're saying is that given a set of malware they was first seen today we seem to be doing better than the AV vendors we've tested against. <-QUOTE}
Correction: First seen today by Prevx. And yes, given this sample set, Prevx does better than the competition. Which is my whole point; it's fairly obvious the graph is meant to convince the average Joe Schmoe that Prevx is vastly superior to the other products shown, but unless a leap of logic is involved, that's actually not what the graph shows at all, due to the biased sample set used.

[19monty64]

{QUOTE-> By the way (to reveal my ignorance even further) -- what the heck is "incremental detection"?:P <-QUOTE}
Sorry bellgamin, nobody seems to want to explain.
By the way, I've seen some good products that have some really bad commercials, but that only convinces me that the commercial is bad-not the product. But then again, I'm not smart enough to know what "incremental detection" is either...

[LUSHER]

{QUOTE-> I use ThreatFire aswell, nice behaviour blocker and a good addition to the Avira scan-based heuristics. Low false positive rate, though I managed to produce a few.

But then, I also have KAV, NOD32, F-PROT, AVG, AVAST and BD installed. Oh and also Process Guard. ;D <-QUOTE}

No No, you might be a AV expert, but you really really need to keep up to date with the innovative security solutions out there today.

Though opions differ on what specific products to use, it is generally understood you should have a setup like below (or you can replace them with alternative products that do the same thing).

1)Antivir - Antivirus
2)Boclean - Antispyware
3)Threatfire - "Behaviorial anti-malware"
4)GesWall - Sandbox
5)Snoopfree - antikeylogger
6)Eqsecure - HIPS "classical"
7)Comodo Memoryguard - Buffer overflow protection
8)Comodo firewall v3 - Software firewall
9) McAffee siteadvisor - warns of malicious sites
10 Retunril virtual system - "Shadow" virtual system

These 8 products represent 10 different and distinct layers that everyone should have. They can be used because they do not overlap in function and uses different approaches to preventing malware and hence each has a role to play.

Add the standard defenses like hosts file, iespyad, spywareblaster, running IE with dropmyrights and running Firefox with no-script/adblock etc, hardening your system with xpsecure, importing ips to blacklist and block with peer guardian etc...

This is way better than running all antiviruses and one outdated Processguard :)

[19monty64]

Some people prefer a "minimalistic" approach and somehow manage to stay malware-free.

[Abeltje]

Yes, let's hope Lusher's post was meant ironically ;)

[Big Apple]

{QUOTE-> Along the same lines, TV commercials dress-up sincere looking actors in white jackets, hang a stethoscope around their necks, & have them give sales spiels for the latest snake oil prescriptions. ;)

By the way (to reveal my ignorance even further) -- what the heck is "incremental detection"?:P <-QUOTE}

Incremental detection? I would assume, that it does something the same as what an incremental backup does. Only backing up, what's changed/added. Detecting incrementally could mean: looking for the newly added/changed stuff, as it remembers what's been scanned already.
This is a pure personal approach and if someone knows exactly......I'd be happy to hear.

[Perman]

{QUOTE-> I use ThreatFire aswell, nice behaviour blocker and a good addition to the Avira scan-based heuristics. Low false positive rate, though I managed to produce a few.

But then, I also have KAV, NOD32, F-PROT, AVG, AVAST and BD installed. Oh and also Process Guard. ;D <-QUOTE}
Hi,

Your post reminds me of a very old event.. back to my teens era.
During art /painting class, a question was popped up " Let mix all those fancy colours, red, orange, purple... and on and on. What colour will this process result in ?" We were all silent suddenly, until someone said " BLACK". A very fundamental and basic shade !

[Montpellier]

{QUOTE-> We were all silent suddenly, until someone said " BLACK". A very fundamental and basic shade ! <-QUOTE}
I was under the impression that if you mixed any number of different colors, you'd always end up with shit brown? ;)

Quite apt don't you think! :D

[aigle]

{QUOTE-> Makes me wonder if Threatfire would even be needed with some AV's like Avira PP or NOD that are known to have outstanding Heuristics for Real Time Detection. <-QUOTE}Yes, it,s needed for sure.

[Diver]

{QUOTE-> Yes, it,s needed for sure. <-QUOTE}

I don't know why you are so sure. Perhaps you can explain why. If McAfee, which is good but not super good, caught 85% of the select sample, it makes me wonder what some of the other higher scoring (on published tests) AV's might do. Then again, Symantec usually beats NcAfee in the same tests, but it only found about 55% of the sample.

[C.S.J]

i have a little question,

i wonder why people moan about ram usage/cpu usage and processes in use, yet still add threatfire to there antivirus, and still complain (mention) about those issues. ::)


lol :wacko:

[solcroft]

{QUOTE-> I don't know why you are so sure. Perhaps you can explain why. If McAfee, which is good but not super good, caught 85% of the select sample, it makes me wonder what some of the other higher scoring (on published tests) AV's might do. Then again, Symantec usually beats NcAfee in the same tests, but it only found about 55% of the sample. <-QUOTE}
Obviously you have a low-risk environment, because you apparently haven't seen how embarrasingly and disastrously AV software fail against new malware on an hourly basis, and your opinion on how well AV software seems to come from third-party tests.

Since that's the case, I'd say no, you probably don't need TF.

[Kerodo]

{QUOTE-> i have a little question,

i wonder why people moan about ram usage/cpu usage and processes in use, yet still add threatfire to there antivirus, and still complain (mention) about those issues. ::)


lol :wacko: <-QUOTE}
ThreatFire adds very little to ram usage (about 7mb total here) and even less to cpu usage (0%) here... So I don't see any problem adding it as a supplement to any existing AV solution...

[ellison64]

Hope im not hijacking the thread ,but ive just installed it after reading some posts here and find that 3 proccesses seem to require internet.(using sygate pro)
1.TF service.exe
2.TFUD.exe
3.TFtray.exe
I can possibly understand the first two requiring access but why would the third?
tia
ellison

[19monty64]

ThreatFire compares applications installed with a community database of trusted apps. and will also investigate what you have allowed as trusted whenever it flags something for suspicious activity. (as well as checking for updates.)

[benton4]

{QUOTE-> ThreatFire compares applications installed with a community database of trusted apps. and will also investigate what you have allowed as trusted whenever it flags something for suspicious activity. (as well as checking for updates.) <-QUOTE}
Are you saying that if others using Threatfire allow bad things, that those things could become part of the 'trusted' list?

[19monty64]

{QUOTE-> Are you saying that if others using Threatfire allow bad things, that those things could become part of the 'trusted' list? <-QUOTE}
No, they are investigated before releasing it to the community. It is similar to the method used by PrevX. The trusted list cuts down on a lot of the false-positives. As with PrevX, no personal identifiers are submitted.

[Diver]

After considering a variety of other enhancements to the standard security precautions of a signature based AV plus two way firewall, what makes Threatfire unusual is that it requires very little, if any, user intervention.

[19monty64]

{QUOTE-> After considering a variety of other enhancements to the standard security precautions of a signature based AV plus two way firewall, what makes Threatfire unusual is that it requires very little, if any, user intervention. <-QUOTE}
Definitely a set-it-and-forget-it app. "Smart-HIPS" is the label I've heard it given.

[Kerodo]

{QUOTE-> After considering a variety of other enhancements to the standard security precautions of a signature based AV plus two way firewall, what makes Threatfire unusual is that it requires very little, if any, user intervention. <-QUOTE}
For me, that is one major selling point...

[likuidkewl]

I have been suing Threatfire on one of my two remaining Windows boxes and it has worked flawlessly with Avast!, this is an old clunker we keep around for emails etc., I have to say with only 384mb of Ram and a Athlon 1100 it brings no noticeable impact to the machine and serves its purpose well(quietly). Great addition to any Windows computer. :thumb: