VBA32 and KAV, same bases ?

[Mack Jones]

Hi,
It's not the first time I see VBA32 using the same name than KAV for a malware...
and if KAV detect some samples VBA32 don't detect (and vice-versa), I wonder if they don't share a part of their bases...
Thanks for your observations gents !

[SDS909]

{QUOTE-> Hi,
It's not the first time I see VBA32 using the same name than KAV for a malware...
and if KAV detect some samples VBA32 don't detect (and vice-versa), I wonder if they don't share a part of their bases...
Thanks for your observations gents ! <-QUOTE}

There are a few strange things comparing them. They both use the IDENTICAL naming conventions, and both seem to have excellent signature detections. Both are Russian companies. In some tests VBA32 seemingly can keep up with KAV on detections (using same naming conventions).

At a security convention here there was a "Rumor" that VBA32 purchased the KAV definition base early on, and spent a great deal of time stripping out the bloat from it. I have no idea if this is true or not, but there is some evidence.

[Mack Jones]

Yes...
And VBA32 seems to be kinda clone of AVP 3.x.
And stranger to see VBA32 detecting malwares KAV don't catch (and vice-versa). Same bases, not the same, partially the same ? ???

[RejZoR]

I doubt they bought bases from them. But they could obtain samples from them (and later add them manually to their own bases).

[IBK]

No, I do not think that KAV shares samples with VBA32. There must be some other unknown reasons.

[.....]

IBK does VBA32 reach the "magical" 85% on your samples yet? Or you not gonna tell us? If it does, will it be included in the tests of 2006?

[IBK]

No, does not reach the 85%. And I predict that it will also not reach in Feb2006, so it will be not included in the on-demand tests next year (but maybe in the retrospective tests; i have to change a bit some rules).

[.....]

Thanks for that.
Is it mainly DOS samples it misses (or can you not say that?)

[likuidkewl]

In my limited selection, it has mainly been Dos nasties that VBA32 misses, and a few ad-related backdoor's/Trojans.

note: Before anyone asks, as I stated before I cannot rightfully test VBA32 and produce non-biases results, due to the fact that as I weed through my small collection of about 15,000 I send them to VBA32.

HTH

[IBK]

{QUOTE-> Thanks for that.
Is it mainly DOS samples it misses (or can you not say that?) <-QUOTE}

No, it is not due the DOS samples; it is low in all categories. Do not ask for more details, I tested it only for the companies.

[mrhero]

{QUOTE-> Hi,
It's not the first time I see VBA32 using the same name than KAV for a malware...
and if KAV detect some samples VBA32 don't detect (and vice-versa), I wonder if they don't share a part of their bases...
Thanks for your observations gents ! <-QUOTE}

In my opinion they only use the name of malware. Here is a sample that VBA32 found but KAV not.
jotti1.jpg

[likuidkewl]

They name straight viruses different.
VBA32
http://www.wilderssecurity.com/supportfiles/p544602-i1.gif

KAV5
http://www.wilderssecurity.com/supportfiles/p544602-i2.gif

[.....]

{QUOTE-> No, it is not due the DOS samples; it is low in all categories. Do not ask for more details, I tested it only for the companies. <-QUOTE}
Many thanks for the info.

[jlo]

And another example of KAV and VBA detecting something with different names.

Cheers

Jlo

Last file scanned at least one scanner reported something about: Server.exe, detected by:

Scanner Malware name
AntiVir X
ArcaVir X
Avast X
AVG Antivirus X
BitDefender X
ClamAV X
Dr.Web X
F-Prot Antivirus X
Fortinet X
Kaspersky Anti-Virus Backdoor.Win32.Amitis.143
NOD32 X
Norman Virus Control X
UNA X
VBA32 Backdoor.Amitis.1

[Mack Jones]

Don't you think VBA32 is too good to detect malwares as well as KAV ?
I can't imagine they don't use (partially) the same bases...
This compagny seems to be too little to perform like this.

[mrhero]

One more example :
jotti3.jpg

[SDS909]

{QUOTE-> No, it is not due the DOS samples; it is low in all categories. Do not ask for more details, I tested it only for the companies. <-QUOTE}

That is strange, I ran VBA32 over 61,000 samples this month and VBA32 scores 94.6%, while Kaspersky scores 96.1% on the same subset.. But my samples are generally Win/Win32 samples, I have zero interest in DOS detections - does anyone?

I'm rather dubious of any claims VBA32 scores less than 85%, especially given its incredible performance at Jotti(and anyone can refresh to see VBA32 detecting stuff nothing else does), and my personal test experiance with VBA.

PS: on those 61,000 samples, the vast majority have identical naming conventions between VBA32 and Kaspersky. That has yet to be explained to me, but is most mysterious.

[SDS909]

{QUOTE-> In my limited selection, it has mainly been Dos nasties that VBA32 misses, and a few ad-related backdoor's/Trojans. <-QUOTE}

I agree. This is why I purged all DOS samples from my inventory - why would I care how a AV performs on DOS? Who the hell runs dos anymore anyway? VBA32 performance on DOS samples is lackluster, while i've witnessed better than KAV performance in many other aspects.

Not to mention I think VBA32 is light and reasonably bug free. Which is difficult to say for some other AVs.

[Firefighter]

{QUOTE-> I'm rather dubious of any claims VBA32 scores less than 85%, especially given its incredible performance at Jotti... <-QUOTE}I agree. They just can't be mostly False Positives or Crap files that Vba32 detects but the most REAL ItW stuff, especially trojan like nasties, where Vba is among the top of scanners. It just seems to be the "wrong" top 3 av:s, that are now in Jotti's according to many here at Wilders. ;D

Best regards,
Firefighter!

[SDS909]

{QUOTE-> I agree. They just can't be mostly False Positives or Crap files that Vba32 detects but the most REAL ItW stuff, especially trojan like nasties, where Vba is among the top of scanners. It just seems to be the "wrong" top 3 av:s, that are now in Jotti's according to many here at Wilders. ;D

Best regards,
Firefighter! <-QUOTE}

This brings up another point - doesn't VBA32 score high for you in your samples Firefighter? Similar to what it scores for me?

[Firefighter]

{QUOTE-> This brings up another point - doesn't VBA32 score high for you in your samples Firefighter? Similar to what it scores for me? <-QUOTE}Maybe even too high. It scored the second, very close to KAV engined av:s and with a few samples more than the third, BitDefender 8.0 Free/9.0 Std, against my 2699 randomly picked nasties collection. 8)

Best regards,
Firefighter!

[Blackcat]

{QUOTE-> No, does not reach the 85%. And I predict that it will also not reach in Feb2006, so it will be not included in the on-demand tests next year). <-QUOTE}
BUT if they are not one of the chosen AV's tested how can they obtain the missed samples? Catch 22!

[IBK]

They get them also from their sources (e.g. other AV companies) or somewhere else, like I do. The reason why companies get missed samples from me is primary not because they do not have already those samples (they usually have them already), it is just in order that they can see that the results are not invented/faked/biased. Due the conditions of the av companies, I am allowed to send missed samples only in accordance with the test conditions and if all conditions are filled.
But as I said, I am planning to include them probably in the other tests (FP and retrospective, where the results may be interesting).

[Firefighter]

{QUOTE-> The reason why companies get missed samples from me is primary not because they do not have already those samples (they usually have them already)... <-QUOTE}Maybe a stupid question but anyway. If those av-vendors already have those missed samples, why do they not add them to their definions then? :-[

Best regards,
Firefighter!

[IBK]

Overload of work? Other priorities? no interest to add them? dunno...
well, they add samples, but I heard e.g. from a company that they have a backlog of 100.000 samples still to check :o , and adding all those takes time (esp. if the company is small and does not have a lot of analysts etc.), so they will first give priority to single samples they get from users (or are currently 'really' ITW), then small collections they get from users, than if time remains and the samples are still undetected, they will samples coming from big collections of other vendors or testers (IMO).

[SDS909]

Either i'm misunderstanding something, or your tests seem biased. Are you saying you provide samples to a "Chosen Few" companies, and the rest are left to fend for themselves?

If this is the case, then I must summarily discount your entire test results in my own mind.

I've tested VBA32 to be 95%+ effective against Win/Win32 threats. The only thing I can't do is certify them (yet). Firefighters results seem identical to mine, yet I have no clue what his random samples contain... Also i've been appraised of a test where VBA32 was sampled against a large set of threats (80k+), and scored 96% on that as well. I'm also privy to Jotti showing VBA32 as extremely superior in detection.

Is something amiss or am I just misunderstanding what is going on here?

[.....]

IBK, tests against samples of 400000+ samples. ;)

[IBK]

bah, again you compare 'Jotti test' (or your tests [no offend intended] with av-comparatives). Do not worry, nothing is wrong with av-comparatives. VBA32 has access to jottis samples. first thing they do, is adding everything they get from jotti every day; those companies that add faster the things uploaded on jotti will look better for those users that visit jotti and upload the same file e.g. the next day, because it will show it at detected. but this does not mean that if you use a very large test-set of malware (i do not speak about spyware etc.), a scanner would detect most things like the things uploaded on jotti. also vba32 has very sensitive heuristics, if even with them enabled to the maximum they actually do not reach 85% on my test-set, i hardly doubt they will in 6 months; maybe in 12 months. another thing i often read is that you say scanner xyx would detect all 'real' itw samples (i wrote 'real' because some here use this term with their own definition) but in reality most of that scanners would usually not even reach 100% (or 99%) of detection of the ITW samples according to the wildlist.
BTW, I could also say: e.g. VBA32 detected here from the av-comparatives test-set at last e.g. 330.000 samples, so why if you used only 80.000 it did not reach 100% in your test? (you see, non-sense question) ;). As I said, if I would look on Win32 threats only, the score would be even lower. But now please stop to get from me details of the VBA32 test results - they are not published and not for public discussion. ;D VBA32 heuristic is cool, they will probably score good in the retrospective test.

back to the topic: VBA32 and KAV, same bases ?
no, they do not use the same bases. vba32 (and also some other companies, e.g. ikarus) looks which scanner does already detect the sample, choose a name of the proposed ones (and as kav naming sceme is 'nice' and kav detects most things) most names they use are similar or the same as those from kav, and some other names will be the same as some other scanner use. my 2 cents...

[Blackcat]

{QUOTE-> Also vba32 has very sensitive heuristics, if even with them enabled to the maximum they actually do not reach 85% on my test-set, i hardly doubt they will in 6 months; maybe in 12 months. <-QUOTE}
IBK, can I ask how you can make this prediction about VBA32 or in fact any of the AV's that you test? Do you have a crystal ball :D ?

[IBK]

sure. i know more or less how much other companies are able to improve in a given period, and by the actual score of VBA32 i guess that they will not be able to improve that much within 6 months. but who knows, maybe they will surprise us all...

EDIT: the "sure" refered to "can i ask", not to "crystal ball", whcih was added later ;)

[Blackcat]

{QUOTE-> sure. i know more or less how much other companies are able to improve in a given period. <-QUOTE}
Can you expand on this? Are you constantly testing the selected AV's against your malware collection?

I would have thought it would be difficult to predict 12 months ahead particularly if a Company decides to add an extra large amount of malware definitions to their database.

Conversely if a particular AV has dropped in detection is this a reflection of them slowing down in adding malware to their database? Or simply a slight change in the malware composition of your collection?

[IBK]

Yes, they are constantly watched. In the test reports there is always a simplified overview of added samples, which is maybe of interest for you.
The reason is probably the first one you said.
Other thing I can also already guess: next year most AV will score very good (due the new versions they will release and due the improvements they are going to add); but it is only a guess...
for august 2005, the results that i am going to release soon are valid and show how they performed the 5th august on the test-set of the 5th august.

P.s.: i saw your otehr post about drweb; do not worry, i am quite sure they will be able to reach easily again advanced in february2006. (just a guess as always). [p.s. sometimes my guesses are wrong, for example i always try to guess how the scanners will perform before i start the tests, but so far i was never able to predict exactly the ranking - so I usually enjoy the surprise of the results too ;)]

[err...]

@IBK
{QUOTE-> bah, again you compare 'Jotti test' (or your tests [no offend intended] with av-comparatives). Do not worry, nothing is wrong with av-comparatives. VBA32 has access to jottis samples. <-QUOTE} I think, that was not the question. The question was, if VBA32 (and the other too poorly scoring AVs) get your testset samples in order to improve.
It seems unfair to give your collection only to the "better" competitors, while the underdogs don't have a (realistic) chance to keep up.

[IBK]

I am sorry, if you find the condition unfair, you have to talk with the av companies (well, not you, the av company that finds it unfair should discuss it with the other av companies). it is not in my hands what the various av companies do with their samples.
btw: i do not find it unfair. i find it absolutly fair and ok so far.
btw2: the sentence that vba32 has access to jottis samples was not in relation to the previous sentence; it was to explain the next following sentence etc.
btw3: what u (or FF and anyone else) can do is to send your samples to the av companies.

[SDS909]

{QUOTE-> IBK, tests against samples of 400000+ samples. ;) <-QUOTE}

and 300,000 of those are Dos viruses. Your point?

[SDS909]

{QUOTE-> I am sorry, if you find the condition unfair, you have to talk with the av companies (well, not you, the av company that finds it unfair should discuss it with the other av companies). it is not in my hands what the various av companies do with their samples.
btw: i do not find it unfair. i find it absolutly fair and ok so far.
btw2: the sentence that vba32 has access to jottis samples was not in relation to the previous sentence; it was to explain the next following sentence etc.
btw3: what u (or FF and anyone else) can do is to send your samples to the av companies. <-QUOTE}

No, it isn't fair, and it *IS* in your hands... You are slanting and biasing your test in favor of certain specific AVs. As a supposed "AV Expert", you yourself should know that this nearly completely invalidates the results of your testing!

I cannot begin to fathom the ridiculousness of providing samples to "Select" AV companies, and then saying "The others will just have to find them their own way".. Worse, you are doing harm to companies, and seem to not be concerned with the "Financial" impact of your published test on the companies that aren't priveledged to recieve your samples..

I have to ask, what do these "Special" companies have to do to recieve your care packages? Pay you? Certainly you wouldn't be the first "Hobbyist" AV tester to have been exposed as biased, or worse....

At the very least, my examination of your tests and proceedures seems to have exposed a specific and admitted bias - a conspiracy if you will, and I don't like what you are doing one bit.

[IBK]

uff, how can i discuss with you about a thing if you do not have a clue about this things (and do not start to offend me if you do not know about those things).
believe me it is fair. they that participate do not get anything what they do not have already. those that do not participate and feel that they do not have those samples have to ask the other av companies if they can get the samples from them (if they do not get already), they can not get it from me. and if they are not allowed to get them from av companies, there will have good reasons for that (no, not financial reasons or because of concurrence), but you seem to be not aware of that.
i think that _you_ are doing harm with your homemade tests to companies and users, as you do not give the chance to check if your samples set is valid or not.
BTW: why should they pay me for something they already have? i of course do not get anything for that.
i explained now everything. 'nice' to see that i get attacked for _no reason_ from a hobby av tester just because i was so friendly to say "No, does not reach the 85%." maybe i should not tell to the public everything what could be of interest, and be inactive in forums like most other testers does... , because even if i explain everything, someone always will not understand it and get it wrongly (and someone can get everything wrongly if he wants to).

[Firefighter]

{QUOTE-> VBA32 has access to jottis samples. first thing they do, is adding everything they get from jotti every day; those companies that add faster the things uploaded on jotti will look better for those users that visit jotti and upload the same file e.g. the next day, because it will show it at detected. <-QUOTE}We have seen somewhere even here at Wilder's those detecting rates about av:s that are included in Jotti's. If I remember right, the top 3. were:

1. Kaspersky

2. Vba32

3. DrWeb

Have I missunderstood something about Jotti's statistics? I understood, that ONLY those FIRST SCANNED SAMPLES were within in Jotti's statistics, not those that were scanned again.

It may also be true that some scanners have not the best available settings to detect malware in Jotti's, but even I have found that Vba32 installed on my PC with best possible heuristics and adware scanning option ON, detected some files that Jotti's Vba32 couldn't, but still Vba32 was the second best there in Jotti's. :-[

Best regards,
Firefighter!

[IBK]

Firefighter, I think you know what my personal hidden opinion is: every test (also the most worst one) does show something and can be of interest, as far as the one who interprets it knows how to weight it and to understand what it shows. I think I posted in past a statistic on how usually the AV on Jotti do score, in order that it is not needed to get an overview by counting only single samples. Your test, jottis site, virusp, etc. they all show something, but they can not be compared with other testing sites, like i can not compare av-comparatives with virusbulletin, checkvir, icsa, etc. (they show all a bit different aspects in some way).

EDIT: found it (the usual results of jotti scanner):
Kaspersky ~83%
VBA32 ~65%
BitDefender ~63%
Dr.Web ~63%
NOD32 ~56%
AntiVir ~54%
ArcaVir ~52%
Fortinet ~48%
ClamAV ~40%

[.....]

{QUOTE-> and 300,000 of those are Dos viruses. Your point? <-QUOTE}
But ~180 000 of them are non-Dos and other OS ::)

[IBK]

and the DOS category is the category where VBA32 scores best in its results (compared to the other scores of VBA32 in the otehr categories) ;)

[BlueZannetti]

At least when I read some of the provisos in the latest www.av-comparatives.org (http://www.av-comparatives.org/) report, the situation seems quite fair. The specific language states that: Vendors supplying samples will receive missed samples after the test if they give permission to allow their samples to be shared with other participants
Those not supplying samples with receive up to 2,500 samples only chosen by the tester
No samples provided and no results listed if detection is below 85% on the on-demand test zoo samples
This seems exceedingly fair. To get something, you have to give something, and that something must be of value (this is where the 85% floor comes into play).

Blue

[err...]

@IBK
{QUOTE-> believe me it is fair. they that participate do not get anything what they do not have already. <-QUOTE} That is imho a critical point. Can you really gurantee that? How can you be sure? Above (post #23) you just said, "they usually have them already". Now what, "usually" or "generally/always"?

Furthermore: do those companies, which do not fulfill your conditions, at least get a chance to know which samples they missed in order to get more than 85%?

I hope you don't mind me asking - but I am sure you understand, that we (your "audience") are not satisfied with a simple "believe me it is fair" ... ;)


@BlueZannetti
{QUOTE-> This seems exceedingly fair. To get something, you have to give something, and that something must be of value (this is where the 85% floor comes into play). <-QUOTE} Well, "fair" in terms of getting/giving... perhaps.
BUT in terms of an objective, fair and non-biased AV test... not really... :(

[BlueZannetti]

{QUOTE-> @BlueZannetti
Well, "fair" in terms of getting/giving... perhaps.
BUT in terms of an objective, fair and non-biased AV test... not really... :( <-QUOTE}The discussion has meandered somewhat off the nominal target of this thread, so if it looks to continue on these lines, that discussion should be split off and continued as a separate topic.

That noted, any test of capability, be it an AV-test or a high school mathematics exam administered to a nervous student, is a partial snapshot.

Good tests provide as objective a metric of capability as feasible. In my personal estimation, the workers at www.av-comparatives.org (http://www.av-comparatives.org/) do an incredible job at getting closer to that ideal than anyone else than I know of.

Save for all vendors unilaterally depositing there samples into a public repository for anyone to sample - and this will never happen since at that point one is, by definition, funding your own competitors - this seems as close to ideal as possible.

If anyone else has a better approach, I would suggest they start a new thread, outline it in detail, and explain how it would function in a commercial environment.

Blue

[IBK]

I have to say "believe me" because it is easier than trying to explain you things that only the ones responsible for the virus labs can understand and know. with usually i mean "nearly all samples".
those that do not get 85% are a) not included in the test and i start now to think that i should in future not again make public test results of a non-included product b) they get more detailed results and if they ask even more detailed than they got (but will cost me more time to prepare for them the documentation and is not really useful data for them).
Yes, I understand that you ask and want to know if everything is done correctly. If you search a bit you find me also asking about otehr testing procedures of others, because I also am always a bit curious and do not believe everything. the conditions that are here for discussion where introduced by the av companies after long discussions how to make it fair, i am only the one who applies them, that is why i see nothing to discuss here, because i know all reasons and am after that discussion convinced that it is absolutly fair. i know i ask much if i just say believe me, but i can not do more for you. of course you are free to not believe me, even if that makes me sorry because i really put all efforts to deliver fair procedures and results.
btw, i know more or less (i do not have all pgp keys) who get which samples from who and who get them again also from me after the test, that is why i know that the published comparatives are fair. who u should not believe to much are some av companies with statements like "we detect all known viruses" or that say "we do not score good in this test because the tester is an #*?§! and did not send us samples" (yes I heard some rumor that someone said this in past, and the most interesting thing is: THEY GOT ALL SAMPLES!, but they do not want to say publicly that it is their fault, so they say the test is shit; usual behaviour); also you should not believe to some reviews you find on the net, last year i find out that some journalists do work not only for magazines but also for some av companies, which did shocked me a bit (for that and also for otehr reasons). So, it is ok if you do not trust everything you read, but I think you can trust me (currently the tests are free of fee, only donations are accepted) and no company has influence in the test results.

[SDS909]

{QUOTE-> i think that _you_ are doing harm with your homemade tests to companies and users, as you do not give the chance to check if your samples set is valid or not. <-QUOTE}

You sir are mistaken, i'm not a "homemade" tester. I do this for a living. I have the faculty to evaluate samples myself, under rare circumstances if I can't, I have people that can and do.

{QUOTE-> Well, "fair" in terms of getting/giving... perhaps. BUT in terms of an objective, fair and non-biased AV test... not really... <-QUOTE}
That is exactly my point.

What I do not like is IBK seems to have a general disdain for people he views as "Hobbiests", and doesn't like sharing information and discounts as "You are a hobbiest, you don't understand.". IBK says he doesn't get paid, so that tells me that he himself is indeed is a hobbiest, possibly masquerading as a professional test organization. Since he has no certification, publishes no industry trade papers, then we must assume our assumptions correct.

No need to beat a dead horse, but I think at the least, Av-Comparative results should be treated with a skeptical eye as are all hobby tests.

[IBK]

{QUOTE-> You sir are mistaken, i'm not a "homemade" tester. I do this for a living. I have the faculty to evaluate samples myself, under rare circumstances if I can't, I have people that can and do. <-QUOTE}

let me guess, you work for an av company. i also am able to evaluate samples by myself, and yes, i also have peoples here that are better than me in analyzing samples and do that.
just because you get paid, it does not mean that your tests are better or more fair. i am not masquerading anything, and if you have problems with av-comparatives, say "you assume" have them and not "we assume".
about no certification: i was already in various visited av companies where i showed that i am able to do the work.
about papers: currently i am quite busy, but i and my collegues are planning to making some papers for virus bulletin in future. i will also be at the virus bulletin conference this year, if you want to discuss with me there, you are welcome.
btw, show me your certification and your papers, i do not see anything from you. i only know that you were an av distributor in past, but no more, that is why i had to assume you are a home av tester. and it is not true that i do not help home av tester, you can ask FF, i helped him a little bit to clean up his collection in my free time.

[Honyak]

I have to admit that while I always appreciated and looked forward to the results of IBK's tests, in the following thread http://www.wilderssecurity.com/showthread.php?t=92147&page=1&pp=25
It appears that IBK was speaking on behalf of Eset until Don Pelotas called this to his attention.
But none the less I appreciate your efforts.

[IBK]

that was a misunderstanding, it tought it was cleared in the meantime; i was defending tests not products. i did not spoke on behalf of anyone.

[Don Pelotas]

{QUOTE-> I have to admit that while I always appreciated and looked forward to the results of IBK's tests, in the following thread http://www.wilderssecurity.com/showthread.php?t=92147&page=1&pp=25
It appears that IBK was speaking on behalf of Eset until Don Pelotas called this to his attention.
But none the less I appreciate your efforts. <-QUOTE}
Actually, that did not come out well and i have apologized to Andreas in a PM conversation about this, so now the apology is official!

I did think think some could see his posts as he was pro-nod (& i mean seen as, & not that he actually was in their corner!), but the way i went about it was perhaps not the best (never post late at night, when brainfog has set in ;) ).

Just for the record i like Andreas's test's, there are not many good ones around with a large number of signatures. :)

[SDS909]

{QUOTE-> btw, show me your certification and your papers, i do not see anything from you. <-QUOTE}

What purpose would this serve? I don't publish test results on a web page, and I don't offer anything more than some basic opinions based on my findings in our day to day operations. (and my opinions are purposely vague) Therefore my creditials are totally irrelevant to this discussion because i'm not soliciting for affirmation of any test or result.

You on the other hand, publish results on a web page, frequent security forums, and seem to have an opinion of yourself as the consumate antivirus expert. As such, i'd expect to see a detailed list of credentials, certificates, training, and a background working in the industry. Otherwise, you should simply label your tests for what they are - hobby testing - and a disclaimer pointing this out along with the test.

This isn't an attack on you personally, i'm merely pointing out your own admitted discrepencies and biases, and the fact that you are a hobby tester, and your test results should be considered with these things in mind. Nothing more, nothing less.

[IBK]

Which bias?? seems like you do not WANT to understand it, so i will stop to discuss with you about this, as you will always start the same. the av companies accredit me (and being accepted by av companies is not something everyone can get). so i do not understand why you start to attack us, are you maybe unhappy that your "kobra tests" are not recognized by the av community?

[Stan999]

Just typical and unfortunately expected sour grapes reactions when someone's current favorite AV didn't do well on a test.

I, for one, appreciate the considerable amount of effort, time and expense Andreas Clementi provided in producing the results and additional details for this On-demand comparative using a large number of signatures.

[Firefighter]

{QUOTE-> VBA32 has access to jottis samples. first thing they do, is adding everything they get from jotti every day; those companies that add faster the things uploaded on jotti will look better for those users that visit jotti and upload the same file e.g. the next day, because it will show it at detected. <-QUOTE}It was new to me that those Jotti's statistics were from you, sorry! But maybe you still missunderstood what I meant. In my attached picture from certain Jotti's scan, there is written:

> (Note: this file has been scanned before. Therefore, this file's scan results will not be stored in the database)

I thought that those detecting rates in Jotti's were based on this assumption just mentioned above, nothing else. If this still is true, I think that those stats looks like that because the top 3 or 4 av:s in Jotti's have also the top update frequencies per year, so they'll add the newiest nasties first which are scanned by HOME users and not by Corporate specialists as the VirusBulletin adds their samples first.

Best regards,
Firefighter!

[IBK]

if the file is e.g. one time uploaded as ZIP or rar file etc. and the second time (e.g. the next day) unarchivied, it will be counted as new file and the results will be displayed (stored in database).
but yeah, your point may be true, probably those that update more frequently will score better there. it may be also true that the scanners which detect more spyware/adware etc. will score better at jotti (as most samples uploaded there are from that categories)

[Firefighter]

{QUOTE-> ... it may be also true that the scanners which detect more spyware/adware etc. will score better at jotti (as most samples uploaded there are from that categories) <-QUOTE}That's true at least when we are comparing VirusTotal against Jotti's. Just checked some Adware samples with DrWeb in VT and Jotti's. They were detected in Jotti's but not in VirusTotal. Maybe VirusTotal doesn't use those risky/nasty beta defs of DrWeb but Jotti's does! :-[

Best regards,
Firefighter!

[SDS909]

This is why I trust VBA32, heavily... I ran across a particularly nasty adware a few minutes ago, i'd go so far as to catagorize it was a trojan downloader. Of course, VBA fired off a warning - good ole' trusty VBA32..

Time and time again, hundreds of weekly samples, VBA32 is the only one detecting this stuff.. This isn't luck, these aren't hand chosen samples, these are real threats on a honeypot machine. 85% detections or less with VBA32? As i've said before, it has never scored less than 95% on anything i've thrown at it, including Zero-Hour outbreak files.

http://www.boredmofo.com/downloads/newthreat9812.JPG
http://www.boredmofo.com/downloads/newthreat9813.JPG

[BlueZannetti]

Just a comment from a distant observer - everyone in the thread is focusing too much on numbers without asking whether the seemingly contradictory results and more casual usage impressions can both be correct. Of course they can!

If you're launching a new AV today, or reaching for wider geographical exposure, covering the threats current and active today is paramount. Your attention will be focused on what people are being exposed to. That is what will make or break you in the word-of-mouth market. Would it be nice to cover all known malware? Sure, but internal resources are limited and you likely have excellent access to currently circulating threats through many of the multivendor malware submission sites on the net. In addition, new users experiencing older malware guarantee that you will be able to hook into legacy malware which is actively circulating as time passes. Net result - excellent field-use detection characteristics.

Say you have an exceptionally comprehensive collection of both zoo and ITW malware. You challenge the AV described above with that collection. Depending on the circulation statistics of the zoo samples, that AV could perform anywhere from admirably to dismally simply because it has potentially been challenged with samples it has never encountered. Net result - detection characterisitics could be anywhere.

So what's this all mean....? Basically understand what the these tests can tell you and what they can't. The tests can confirm excellent performance (the 95+% products are good), on the other hand they do not demonstrate poor/marginal performance in the field (the <85% products are not necessarily bad). For those wishing for black and white delineation, this may be somewhat disconcerting, but this is forced by the comparative nature of the test vs. the real world.

Personally, I see no necessary conflict in a < 85% detected test result and anecdotal field observations which suggest exceptionally good performance. It an almost forced situation for products early in their lifecycle.

Blue

[Tweakie]

{QUOTE-> bah, again you compare 'Jotti test' (or your tests [no offend intended] with av-comparatives). Do not worry, nothing is wrong with av-comparatives. VBA32 has access to jottis samples. first thing they do, is adding everything they get from jotti every day; those companies that add faster the things uploaded on jotti will look better for those users that visit jotti and upload the same file e.g. the next day, because it will show it at detected. but this does not mean that if you use a very large test-set of malware (i do not speak about spyware etc.), a scanner would detect most things like the things uploaded on jotti. <-QUOTE}

Of course, you realize that this scheme also applies to av-comparatives. Since you do provide undetected samples to the companies after the tests, they could decide to add it to their signatures in priority before the next test.

However, there is an easy way of detecting that :
- from the graphs that you published on page 3 of the test report, you can easily compute the improvement rate for each AV.
- Then, you can also compare the improvement rates (between this test and the previous on-demand test) for the malwares that have been added to the test set between two tests (don't forget to remove the malware that have been provided by the vendor and the malware that have been used for the proactive tests, since vendors know that you will use it).

If for a particular AV these improvement rates are too different (i.e. improvement over the (large) already-used test set is higher than for the (smaller) new test set), you can deduce that the AV vendor is cheating a little bit.