Jotti's Tests

In the tests of specific malware that Jotti's shows, I note that one of the best AVs, NOD32, does not seem to have the detection rate of some others. BD appears to be one of the best there.
I also note that KAV misses more than I would expect.

Is there some reason that NOD misses more than BD when the heuristics of NOD are the best?

This is not intended to start a comparison as such, but my impression is that it does not square with AV Comparatives.
I wish there was some record keeping of the Jotti's results so that over a period of time we could see the success of the the various AVs.

I do not question the validity of Jotti's and believe they are independent and that the results are factual.

Jerry

 

Jotti uses the Linux versions of each scanner and all the functions/options that are available on Windows versions are not available on Linux.

 

Jotti doesn't test anything. He probably keeps a track record for himself, but in general it's just a scan service, nothing else.

 

JerryM read this: https://www.wilderssecurity.com/showpost.php?p=747090&postcount=38

Additionally don't forget that NOD32 doesn't support all runtime-packers and some file submitted to Jotti's are not detected due to this fact.

 

Thanks, All, for the replies. That helps.
Jerry

 

I also seem to recall something Marcos mentioned about NOD32's heuristics not working properly at jotti's? Also, you can never know if the software is properly updated or configured (f.ex. I remember a time when BitDefender at jotti's didn't really catch anything at jotti's while BitDefender at VirusTotal did).

Bottom line is, you can't really trust any scan service 100 percent. And it shouldn't be used as a "reliable test site".

 

You won't believe what kind of binary crap (damaged samples, non-malware etc.) people upload to Jotti and VirusTotal - it's little wonder that NOD32 doesn't catch those files.

Simply do not use Jotti and VirusTotal to "measure" the quality of an antivirus product.

 

So how do you measure the quality of such products?

 

watching independent tests like av-comparatives.org, virusbulletin.com and trialing for yourself.

 

In no specific order:

- reaction time for ITW and non-ITW malware, zero-day-exploits
- static unpacking (how many, how reliable against small modifications?)
- x86 emulation (handling of highly complex polymorphic/metamorphic malware)
- x86 emulation (generic unpacking, speed, available for all platforms?)
- heuristic (win32, script etc.)
- variant detection
- script handling (emulation? VBA, VBS, JS, see Feebs)
- unpacking of archives (reliable (exploits!), speed, archive/mail bombs, platforms?)
- speed of scanning (clean and infected files)
- detection of exploits (= correct file format parsing & analysing)
- how exact is the malware identification (if the product does bother to identify at all)
- memory usage of the scan engine
- size of the pattern updates, size of the engine updates
- behaviour blocker (false positives? end-user-friendly?)
- handling of active malware in memory
- cleaning of infected systems (infections with multiple malware handled?)
- same detection quality for on-demand and on-access
- system load of on-access scanning
- stability (there were lots of exploitable weaknesses in AV archive handling)
- available for many platforms?
- outbreak detection (e.g. on gateways)

Alas, there is no published test that measures all these things :-(

 

Not to mention ease of use, customer satisfaction, tech support, engine vulnerability, upgrade frequency (not update frequency), price etc. etc.

 

I 2nd that lots of crappy files are uploaded to Jotti's - far much more than to VT. What's more, there are some problems with AVs allocating too much memory at Jotti's which results in that some scanners do not even scan particular samples, and they are subsequently shown as undetected by these AVs. It is also common that the samples submitted are actually installers which are not detected, but the malicious files are detected upon extraction.

 

I always notice VBA32 picking up everything sent to Jottis. VBA32 seems to detect most any real world stuff I run into as well, so I am pleased to use the product.

 

Quoting Jotti "You're free to (mis)interpret these automated, flawed statistics at your own discretion. For antivirus comparisons, visit AV comparatives".

 

AV Comparatives is the only company, which public "independent" tests, who knows if they are accurate, when there are no other tests to compare? Jotti is Linux based, but it is testing windows files, and if some AVs even does not have signature for it, at least it shows their reaction time, which is important for zero day exploits. Anyway, any known AV is good.

 

If an AV flagged all files on your disk, including malicious files, would it actually be the best just because it detected all threats?

You must distinguish between legit and malicious files, functional and non-functional files, archives/installers and unpacked files, etc.

Regarding Jotti's results, many scanners even don't get to scanning as the previous scanners have allocated too much memory. Sometimes submitting an undetected file again will show it as detected.

 

Simply do not use any comparative results that do not fit into your vision of the univers. Just use the product you like

 

seems to be Severyanin day-by-day practise (just kidding)

 

Its hard when somebody do not want to change his mind no matter if there are arguments or not

By my expirience with Jotti and VT the following scanners have good detection:

- DrWeb
- Kaspersky
- NOD32
- VBA32
- AntiVir

All others miss a lot.

 

Besides the corrupted samples that makes such multi-online scannes are not so reliable and probably has not thing to do with real-world capability of AV scannes. I'd say the scanners itself are also the important factor that make such results are not so reliable, because of some scannes are the older version compared to their latest desktop/server engine version, so this causes a different results.

For example, the version of avast! engine on VirusTotal is 4.6.695 but the official latest version of avast! engine for desktop is 4.7.827. I have some malware sample that my avast! on desktop detect but avast! engine on VirusTotal does not and I'm sure there're lots of such sample in my malware collection.