Heuristics in action

Saraceno's thoughts echo some of my long held beliefs.

AV-Comparatives listing 21 FP's as "many FP's" rather than "few FP's" should be changed, in my humble opinion

Been watching the AVC etc tests for several years, and if they didn't take off points for FP's then from what i've seen, Avira consistantly ranks as tops for detection. And remember, that's with Heuristics/settings not even on max

That's why i'd like to see other tests on AV's to compare how well, or not, they shape up with Heuristics etc fully enabled.

 

If you look at AVC Avira was the ONLY product to lose a rank due to FP's because it detected so many.

And Stefan why would I help a product I dont use?

 

@PC_Gamer & whitedragon:

I'm not quite sure how you two could so completely misinterpret my post.

I said _nothing_ whatsoever about not detecting infected keygens/cracks.

Actually i even said nothing about the detection of cracks/keygens at all?

 

You said that fixing FP's in Keygens was not a priority. What if a legitimate program was packed the same way and was also a false positive because of the lack of research on your companies end? After all Id take a gamble and say that alot of keygens are infact clean they just get reported as viruses, hacktools, etc only because of the way they are packed.

 

You are wildly jumping to conclusions on "what if" scenarios full of unsubstantiated accusations.

If we receive an FP on a _legitimate_ file, we fix it. Period. Send your FPs to Stefan, he'll give them a look.

( continuing vacation now )

 

I agree with WhiteDragon, Many keygens are not infected the way they are flagged... Even only one Antivirus is dare to tell that the "X" keygen is Not.A.Virus. and that is VirusBuster.

 

oops, I misread the February 2009 report as February 2010.

In the August 2009 report Avira reduced the number of FPs but other popular products reduced them more, so Avira had about twice the median. Still Avira was the product with the lowest number of FPs between the products with "many FP's" as usual. Other 5 engines had more FPs than Avira, 4 of them were penalized.

 

That's one of the main problems with AVC's ratings - it is subjective as to how much a penalty to give to one criteria such as detection or FP's when designating a product Advanced Plus or simply Advanced.

For example, in the latest Retrospective test, Avira detected 17,282 viruses/malware vs. ESET's 14,005 and Avira had 21 FP vs. ESET's 12 and ESET is designated Advanced Plus, and Avira Advanced.

I don't know how big the clean file sample in FP testing is (don't see the size published anywhere), but I'd much rather detect thousands more viruses/malware and live with a few more FP's.

 

I said it previously, and I'll say it again.

Compared to other AVs:
Let's say, for every one false positive it will give you, Avira will protect you and detect 1000 more trojans and viruses the other leading security product will miss.

Compared to the one located right next to Avira in AV-C's report:
"Let's say, for every one false positive it will give you, Avira will protect you and detect 309 more trojans and viruses the other leading security product will miss."

All this talk of keygens. Many were quick to frown upon a forum that distributed keygens, but using keygens is ok? Is software created from the clouds and magically appears on the internet and your machine? Keygens are activating software without purchasing. More simply, getting something for free you haven't paid for.

I don't mean to sound all high and mighty. But it's ok to talk about keygens and accept that's what many people are into, but on this forum which has many developers spending their valuable free time to help others while trying to pay their own personal bills, talking about 'clean keygens' or any keygens whatsoever is actually distasteful to all of them (software developers).



Edit - maybe I'm getting too old. Don't mind me...

 

Its unfortunate that those software developers brought it up. If their product cant tell a clean file from a infected one because of the way its packed its not my issue.

On the other hand we couldnt find any info on the size of the collection of infections that AVC tested. If they tested 50 files and Avira had 21 FP's then thats a huge issue., but we dont know the size of the collection that was used so its malinformed to say:

For every FP in Avira it protects against [insert random number here] detections over the next best.

Theres simply no proof of that.

 

I agree with you that its really distasteful for the software developers, but i hardly found any other software developer here except some security software developers and "Paragon"...But i do agree that we should not use these keygens becuase it will directly hurt the real developers....

But what we are trying to discuss here is that AV companies should not flag non-malicious files as a malware because of their so called packers. Even if it is keygen they should not be marked as malware...they can be marked as HackTool.Keygen or something else which can clearly indicate user what the file is?

Even we have saw that VirusBuster is using this kind of Nomenclature to keygens and patches which are clean or non malicious...then why not other companies dare to do this...

 

True, you don't know how many safe files were used in the test.

But the other results, I used the last AV-C report (main test - august/november) which states the total number of malicious files, the number of detections, and the appendix which states the number of false positives. Based on their test, and the results given, you can generate the stats and Avira was the best AV by far. Almost doubled the detections of some products which received the same rating.

Avira detects 17 282 out of 23 237 malicious files. Awarded advanced.
Norton detects 8465 out of 23 237 malicious files. Awarded advanced.

Avira doubled its detections. But 8 more false positives for Avira brought them on the same level.

Edit : AvinashR, all ok, as it's a current issue, so I respect that. Just thought I'd speak my mind eg. Prevx, Defense Wall, MBAM, Malware Defender, Avira, Sunbelt, Hitman Pro, Dr Web, Avast and others read here .

 

So the last test was 23,237 samples? I wonder why they opted to not publish sample size for the latest test. Perhaps thats some insight that the sample size has diminished and they dont want to spill the beans. Seems rather shady to me.

 

There isn't any direct linking to reports, so it's difficult to provide the exact link. But the tests I'm referring to are:

AV-C main site. Comparatives/Reviews. Then main tests. Number 24 - Retrospective/proactive test - November 2009. On page 6, has a PDF link to the false positives.

It's good to discuss these things, as I wouldn't have bothered otherwise to look as closely at the results.

 

I looked through the FP report, but didnt see anything mentioned on sample size.

 

I can't find that either. They should list in their next report something along the lines of, 'of the 20 000 malicious files used in the test, we added 2000 safe files'.

Then we could get a better understanding of how many false positives a product generated.

 

Some individuals may look at such statistics and infer that Avira provides superior malware protection. Unfortunately, the inference is incorrect. These statistics are based upon malware detection -- not prevention, which is the real-world outcome users care about, in my opinion. For example, when you examine the whole-product dynamic test conducted by AV-Comparatives (December, 2009), Avira scores well -- but, it does not hold the top position in the rankings.

 

I guess this thread drastically diverted to another hot topic...

 

Valid point Pleonasm, and also people should look at the products tested. For example, is it an AV, or the edition with more enhanced protection.

Not to say Avira is twice another program. Just pointing out the FP system might need some tweaking as Avira's performance in that one test might have been overlooked.

For example, to get advanced, detection rates can be from 25-50 per cent. That is too wide a scope. Advanced should be 40-60 per cent, advanced plus 60 percent and above.

And you're right, many tests show different and varying results. That's all from me, I'm out (ironclad guarantee), back to the topic of heuristics.

 

As long as the false positive is a keygen and not an O/S file that kills my PC, I'll take a few extra FP's any day in exchange for way better detection numbers.

 

Yes, indeed let's get down to semantics yet again: detection versus prevention. Avira detects, and detects perhaps more than it should. Detection for people who haven't used Avira means many possibilities if it is interactive. It gives you many options, namely ignore, delete, quarantine etc. But the main thing is that it detects more (as a single engine) than anybody else. Norton has been doing well lately, but only lately and on one single test. Let's see what the next round of tests will bring us.

People relying on an antivirus alone and believing that it can alone solve their problems, are a bit naive.

 

But, why should anyone care? Such an empirical comparison is nearly irrelevant to the question of which product provides superior protection against malware, where “protection = prevention + (detection & disinfection).”

Detection, as measured by AV-Comparatives in a contrived offline laboratory environment, simply does not reflect the full capabilities of an anti-malware product. Therefore, a detection score is not an indication of true real-world protection, and thus is “academically interesting” but otherwise is of questionable utility, in my opinion.

 

But how did you found out you have insane amounts of false positives on your hard disc if you did not scan with Avira? Uploaded all your executables to VirusTotal?

Excuse me, but I have to call BS on this one. It's easy coming here posting wild claims - and then of course fail to deliver any proof.
I doubt that you have more than 3 false positives on legit programs (which are non-patched/cracked).

 

Stefan, the posts you refer to are unnecessarily confrontational & trollish. They do not contribute to learning or advancement of security issues. Therefore, why do you bother replying to such nonsense? Get back to work on version 10, please.

 

How on earth did this topic get so screwed up? Concentrating on keygen and crack detection? Who truly believes in that nutty idea? It's well known that keygens and cracks have had FP detections for years...and it's equally well known that they often truly are infected. If you use them, and I have in the past, sue me...then you risk infection, it's as simple as that. I for one sure as hell don't expect AV or AM vendors to work on FP detections for that category of software.

These vendors are there to protect you from rampant drive-bys, hijacked websites hosting previously clean versions of legitimate files, and the like. Their job is not to let you wander around the web cracking this or that, purposely downloading malware to play with it and all this crap.