winnow.oitc.com/AntiVirusPerformance

Longboard · Mar 26, 2007

This site has been getting some interesting attention.

http://winnow.oitc.com/AntiVirusPerformance.html

I e-mailed the home page site operators ( http://www.oitc.com/OITC/OITCHome.html )and had a very polite response which I quote with permission:

We are a small consulting company and we have no agenda other than to
see if the statistics support something that most people know but
don't want to admit to and maybe to learn some other things in the
process.

I am amazed at some of the postings on the links you sent. The
funniest ones are the "conspiracy theorists." What's funny is that
there is no "test" per se. This is a display of the statistics coming
from an accumulation of a real-time set of samples. We are using it
to try to understand the real world effectiveness of perimeter
defenses when employing these engines.

We contacted each engine company and asked them how their engines
were configured at Virus Total and what their "output names" actually
meant. Not all answered but most did. We then reprocessed the older
data based upon feedback from the vendors in February to be able to
maintain the cumulative running stats back to the beginning of Dec
(or whenever an AV system was added to VirusTotal). This is the
basis of the statistics displayed.

Its interesting that from a number of comments show that some posters
seem to have never looked at the explanation of the methodology
(http://winnow.oitc.com/avreadme.html) or that they did not check the
underlying data. An example is the comments about Webwasher which has
only been running at VirusTotal since 3/21 and has amassed a total of
161 samples in comparison to almost 2000 samples for most of the
other engines (http://winnow.oitc.com/avmalwarestats.php).
Personally, I would wait a while before assessing Webwasher's real
world performance.

The graph displays exactly what it says it does. There is no bogus
nor bias. Click and look at each engine's statistics. All the warts,
if any, are there for all to see.

We are not interested from a home user perspective (which some
posters seem to be because of comments about manual interaction or
not being worried because they can always roll back), and we are not
interested in how the engines perform in the lab against huge
databases of viruses/malware that have been collected nor are we
interested in how well they can "cleanup a mess."

We are trying, however, to see if we can characterize their real
world performance in detecting early outbreaks of malware at the
enterprise perimeter. To do this, we have started collecting
statistics using fresh malware only, running it through the various
engines( configured for hands off, non ultra paranoid mode) and
collecting statistics until a majority of engines detect the malware.
Then, we stop collection of statistics on that "outbreak." This is
our way of estimating the time for "early outbreaks" determining that
this exemplified the average real world time it takes to for
something to be released, captured, analyzed, dat files updated and
dat files distributed to users. This is how the data is collected and
is then compiled into the statistics displayed.

Shortly, we will be adding a new set of tracking statistics that
looks at the performance one could expect by combined engines at the
perimeter.

Hopefully, we will all learn more as the sample size grows.

I am amazed at the "surprise." No one should be surprised at these
stats (even though there seems to be be a lot of chatter on those two
links you sent me). Well, maybe the ranking but not the fact of poor
performance as near zero day outbreaks are problematic which shows
that one need defense in depth and strict enterprise policies.

I hope this answered your questions.
Click to expand...

Blue Zanetti has a good perspective here:
https://www.wilderssecurity.com/showthread.php?t=169602

Further quoting from reply e-mails from OITC

What's interesting is that with all the buzz, only a senior Panda
researcher and you have contacted me directly about the data and the
stats. Oh well....
If you have any further questions or comments or suggestions do not
hesitate to send them forward.
Click to expand...

So>
Interesting application/analysis of reputable data

Be interesting to see how the service might develop

Regards.

lucas1985 · Mar 27, 2007

It seems reasonable that they are interested in the performance of perimeter/gateway solutions in outbreak situations.
As always, statistics should be looked in perspective.

Log in or Sign up

winnow.oitc.com/AntiVirusPerformance

Longboard Registered Member

lucas1985 Retired Moderator

Log in or Sign up

winnow.oitc.com/AntiVirusPerformance

Longboard Registered Member

lucas1985 Retired Moderator

Useful Searches